{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ec7a55e1",
   "metadata": {},
   "source": [
    "# 文本数据处理"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbaabb36",
   "metadata": {},
   "source": [
    "此部分涵盖了大语言模型的数据预处理部分。课程内容架构如下图所示："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "069ef78a",
   "metadata": {},
   "source": [
    "![](./images/framework.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9656720d",
   "metadata": {},
   "source": [
    "## 理解词嵌入"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cf0cd02",
   "metadata": {},
   "source": [
    "嵌入有多种形式，本课程聚焦于文本嵌入的研究。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1cf9acd",
   "metadata": {},
   "source": [
    "![](./images/embedding.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10afc425",
   "metadata": {},
   "source": [
    "- 大型语言模型处理高维空间中的嵌入；\n",
    "- ​​由于无法可视化此类高维空间（人类思维限于1、2或3维度），下图展现了一个二维嵌入空间​​；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb993b87",
   "metadata": {},
   "source": [
    "![](./images/vector.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "641678f2",
   "metadata": {},
   "source": [
    "本节将对文本进行分词，即将文本拆分为较小单元，例如单词和标点符号​​。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4738b7e",
   "metadata": {},
   "source": [
    "![](./images/pipeline.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a35dd91c-b1bd-4243-a90b-a3c3879121d2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20479\n"
     ]
    }
   ],
   "source": [
    "with open(\"corpus.txt\", \"r\", encoding=\"utf-8\") as f:\n",
    "    raw_txt = f.read()\n",
    "\n",
    "print(len(raw_txt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4797e7af-72c7-4ee1-ab32-258c8b999f66",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I HAD always thought Jack Gisburn rather a cheap genius--though a good fellow enough--so it was no great surprise to me to hear that, in the height of his glory, he had dropped his painting, married a\n"
     ]
    }
   ],
   "source": [
    "print(raw_txt[:200])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9a3c66d8-e5f2-4650-9ba4-02d443bb6f7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f67a7e1c-ef25-4eff-a283-cbce3dca6d01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Hello,', ' ', 'word.', ' ', 'This', ' ', 'is', ' ', 'an', ' ', 'example!']\n"
     ]
    }
   ],
   "source": [
    "text = \"Hello, word. This is an example!\"\n",
    "result = re.split(r'(\\s)', text)\n",
    "\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7262cdbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = re.split(r'([,.:;?_!\"()\\']|--|\\s)', raw_txt)\n",
    "processed = [item.strip() for item in result if item.strip()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3a8e7157",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['I', 'HAD', 'always', 'thought', 'Jack', 'Gisburn', 'rather', 'a', 'cheap', 'genius', '--', 'though', 'a', 'good', 'fellow', 'enough', '--', 'so', 'it', 'was', 'no', 'great', 'surprise', 'to', 'me', 'to', 'hear', 'that', ',', 'in', 'the', 'height', 'of', 'his', 'glory', ',', 'he', 'had', 'dropped', 'his', 'painting', ',', 'married', 'a', 'rich', 'widow', ',', 'and', 'established', 'himself', 'in', 'a', 'villa', 'on', 'the', 'Riviera', '.', '(', 'Though', 'I', 'rather', 'thought', 'it', 'would', 'have', 'been', 'Rome', 'or', 'Florence', '.', ')', '\"', 'The', 'height', 'of', 'his', 'glory', '\"', '--', 'that', 'was', 'what', 'the', 'women', 'called', 'it', '.', 'I', 'can', 'hear', 'Mrs', '.', 'Gideon', 'Thwing', '--', 'his', 'last', 'Chicago', 'sitter', '--', 'deploring', 'his', 'unaccountable', 'abdication', '.', '\"', 'Of', 'course', 'it', \"'\", 's', 'going', 'to', 'send', 'the', 'value', 'of', 'my', 'picture', \"'\", 'way', 'up', ';', 'but', 'I', 'don', \"'\", 't', 'think', 'of', 'that', ',', 'Mr', '.', 'Rickham', '--', 'the', 'loss', 'to', 'Arrt', 'is', 'all', 'I', 'think', 'of', '.', '\"', 'The', 'word', ',', 'on', 'Mrs', '.', 'Thwing', \"'\", 's', 'lips', ',', 'multiplied', 'its', '_', 'rs', '_', 'as', 'though', 'they', 'were', 'reflected', 'in', 'an', 'endless', 'vista', 'of', 'mirrors', '.', 'And', 'it', 'was', 'not', 'only', 'the', 'Mrs', '.', 'Thwings', 'who', 'mourned', '.', 'Had', 'not', 'the', 'exquisite', 'Hermia', 'Croft', ',', 'at', 'the', 'last', 'Grafton', 'Gallery', 'show', ',', 'stopped', 'me', 'before', 'Gisburn', \"'\", 's', '\"', 'Moon-dancers', '\"', 'to', 'say', ',', 'with', 'tears', 'in', 'her', 'eyes', ':', '\"', 'We', 'shall', 'not', 'look', 'upon', 'its', 'like', 'again', '\"', '?', 'Well', '!', '--', 'even', 'through', 'the', 'prism', 'of', 'Hermia', \"'\", 's', 'tears', 'I', 'felt', 'able', 'to', 'face', 'the', 'fact', 'with', 'equanimity', '.', 'Poor', 'Jack', 'Gisburn', '!', 'The', 'women', 'had', 'made', 'him', '--', 'it', 'was', 'fitting', 'that', 'they', 'should', 'mourn', 'him', '.', 'Among', 'his', 'own', 'sex', 'fewer', 'regrets', 'were', 'heard', ',', 'and', 'in', 'his', 'own', 'trade', 'hardly', 'a', 'murmur', '.', 'Professional', 'jealousy', '?', 'Perhaps', '.', 'If', 'it', 'were', ',', 'the', 'honour', 'of', 'the', 'craft', 'was', 'vindicated', 'by', 'little', 'Claude', 'Nutley', ',', 'who', ',', 'in', 'all', 'good', 'faith', ',', 'brought', 'out', 'in', 'the', 'Burlington', 'a', 'very', 'handsome', '\"', 'obituary', '\"', 'on', 'Jack', '--', 'one', 'of', 'those', 'showy', 'articles', 'stocked', 'with', 'random', 'technicalities', 'that', 'I', 'have', 'heard', '(', 'I', 'won', \"'\", 't', 'say', 'by', 'whom', ')', 'compared', 'to', 'Gisburn', \"'\", 's', 'painting', '.', 'And', 'so', '--', 'his', 'resolve', 'being', 'apparently', 'irrevocable', '--', 'the', 'discussion', 'gradually', 'died', 'out', ',', 'and', ',', 'as', 'Mrs', '.', 'Thwing', 'had', 'predicted', ',', 'the', 'price', 'of', '\"', 'Gisburns', '\"', 'went', 'up', '.', 'It', 'was', 'not', 'till', 'three', 'years', 'later', 'that', ',', 'in', 'the', 'course', 'of', 'a', 'few', 'weeks', \"'\", 'idling', 'on', 'the', 'Riviera', ',', 'it', 'suddenly', 'occurred', 'to', 'me', 'to', 'wonder', 'why', 'Gisburn', 'had', 'given', 'up', 'his', 'painting', '.', 'On', 'reflection', ',', 'it', 'really', 'was', 'a', 'tempting', 'problem', '.', 'To', 'accuse', 'his', 'wife', 'would', 'have', 'been', 'too', 'easy', '--', 'his', 'fair', 'sitters', 'had', 'been', 'denied', 'the', 'solace', 'of', 'saying', 'that', 'Mrs', '.', 'Gisburn', 'had', '\"', 'dragged', 'him', 'down', '.', '\"', 'For', 'Mrs', '.', 'Gisburn', '--', 'as', 'such', '--', 'had', 'not', 'existed', 'till', 'nearly', 'a', 'year', 'after', 'Jack', \"'\", 's', 'resolve', 'had', 'been', 'taken', '.', 'It', 'might', 'be', 'that', 'he', 'had', 'married', 'her', '--', 'since', 'he', 'liked', 'his', 'ease', '--', 'because', 'he', 'didn', \"'\", 't', 'want', 'to', 'go', 'on', 'painting', ';', 'but', 'it', 'would', 'have', 'been', 'hard', 'to', 'prove', 'that', 'he', 'had', 'given', 'up', 'his', 'painting', 'because', 'he', 'had', 'married', 'her', '.', 'Of', 'course', ',', 'if', 'she', 'had', 'not', 'dragged', 'him', 'down', ',', 'she', 'had', 'equally', ',', 'as', 'Miss', 'Croft', 'contended', ',', 'failed', 'to', '\"', 'lift', 'him', 'up', '\"', '--', 'she', 'had', 'not', 'led', 'him', 'back', 'to', 'the', 'easel', '.', 'To', 'put', 'the', 'brush', 'into', 'his', 'hand', 'again', '--', 'what', 'a', 'vocation', 'for', 'a', 'wife', '!', 'But', 'Mrs', '.', 'Gisburn', 'appeared', 'to', 'have', 'disdained', 'it', '--', 'and', 'I', 'felt', 'it', 'might', 'be', 'interesting', 'to', 'find', 'out', 'why', '.', 'The', 'desultory', 'life', 'of', 'the', 'Riviera', 'lends', 'itself', 'to', 'such', 'purely', 'academic', 'speculations', ';', 'and', 'having', ',', 'on', 'my', 'way', 'to', 'Monte', 'Carlo', ',', 'caught', 'a', 'glimpse', 'of', 'Jack', \"'\", 's', 'balustraded', 'terraces', 'between', 'the', 'pines', ',', 'I', 'had', 'myself', 'borne', 'thither', 'the', 'next', 'day', '.', 'I', 'found', 'the', 'couple', 'at', 'tea', 'beneath', 'their', 'palm-trees', ';', 'and', 'Mrs', '.', 'Gisburn', \"'\", 's', 'welcome', 'was', 'so', 'genial', 'that', ',', 'in', 'the', 'ensuing', 'weeks', ',', 'I', 'claimed', 'it', 'frequently', '.', 'It', 'was', 'not', 'that', 'my', 'hostess', 'was', '\"', 'interesting', '\"', ':', 'on', 'that', 'point', 'I', 'could', 'have', 'given', 'Miss', 'Croft', 'the', 'fullest', 'reassurance', '.', 'It', 'was', 'just', 'because', 'she', 'was', '_', 'not', '_', 'interesting', '--', 'if', 'I', 'may', 'be', 'pardoned', 'the', 'bull', '--', 'that', 'I', 'found', 'her', 'so', '.', 'For', 'Jack', ',', 'all', 'his', 'life', ',', 'had', 'been', 'surrounded', 'by', 'interesting', 'women', ':', 'they', 'had', 'fostered', 'his', 'art', ',', 'it', 'had', 'been', 'reared', 'in', 'the', 'hot-house', 'of', 'their', 'adulation', '.', 'And', 'it', 'was', 'therefore', 'instructive', 'to', 'note', 'what', 'effect', 'the', '\"', 'deadening', 'atmosphere', 'of', 'mediocrity', '\"', '(', 'I', 'quote', 'Miss', 'Croft', ')', 'was', 'having', 'on', 'him', '.', 'I', 'have', 'mentioned', 'that', 'Mrs', '.', 'Gisburn', 'was', 'rich', ';', 'and', 'it', 'was', 'immediately', 'perceptible', 'that', 'her', 'husband', 'was', 'extracting', 'from', 'this', 'circumstance', 'a', 'delicate', 'but', 'substantial', 'satisfaction', '.', 'It', 'is', ',', 'as', 'a', 'rule', ',', 'the', 'people', 'who', 'scorn', 'money', 'who', 'get', 'most', 'out', 'of', 'it', ';', 'and', 'Jack', \"'\", 's', 'elegant', 'disdain', 'of', 'his', 'wife', \"'\", 's', 'big', 'balance', 'enabled', 'him', ',', 'with', 'an', 'appearance', 'of', 'perfect', 'good-breeding', ',', 'to', 'transmute', 'it', 'into', 'objects', 'of', 'art', 'and', 'luxury', '.', 'To', 'the', 'latter', ',', 'I', 'must', 'add', ',', 'he', 'remained', 'relatively', 'indifferent', ';', 'but', 'he', 'was', 'buying', 'Renaissance', 'bronzes', 'and', 'eighteenth-century', 'pictures', 'with', 'a', 'discrimination', 'that', 'bespoke', 'the', 'amplest', 'resources', '.', '\"', 'Money', \"'\", 's', 'only', 'excuse', 'is', 'to', 'put', 'beauty', 'into', 'circulation', ',', '\"', 'was', 'one', 'of', 'the', 'axioms', 'he', 'laid', 'down', 'across', 'the', 'Sevres', 'and', 'silver', 'of', 'an', 'exquisitely', 'appointed', 'luncheon-table', ',', 'when', ',', 'on', 'a', 'later', 'day', ',', 'I', 'had', 'again', 'run', 'over', 'from', 'Monte', 'Carlo', ';', 'and', 'Mrs', '.', 'Gisburn', ',', 'beaming', 'on', 'him', ',', 'added', 'for', 'my', 'enlightenment', ':', '\"', 'Jack', 'is', 'so', 'morbidly', 'sensitive', 'to', 'every', 'form', 'of', 'beauty', '.', '\"', 'Poor', 'Jack', '!', 'It', 'had', 'always', 'been', 'his', 'fate', 'to', 'have', 'women', 'say', 'such', 'things', 'of', 'him', ':', 'the', 'fact', 'should', 'be', 'set', 'down', 'in', 'extenuation', '.', 'What', 'struck', 'me', 'now', 'was', 'that', ',', 'for', 'the', 'first', 'time', ',', 'he', 'resented', 'the', 'tone', '.', 'I', 'had', 'seen', 'him', ',', 'so', 'often', ',', 'basking', 'under', 'similar', 'tributes', '--', 'was', 'it', 'the', 'conjugal', 'note', 'that', 'robbed', 'them', 'of', 'their', 'savour', '?', 'No', '--', 'for', ',', 'oddly', 'enough', ',', 'it', 'became', 'apparent', 'that', 'he', 'was', 'fond', 'of', 'Mrs', '.', 'Gisburn', '--', 'fond', 'enough', 'not', 'to', 'see', 'her', 'absurdity', '.', 'It', 'was', 'his', 'own', 'absurdity', 'he', 'seemed', 'to', 'be', 'wincing', 'under', '--', 'his', 'own', 'attitude', 'as', 'an', 'object', 'for', 'garlands', 'and', 'incense', '.', '\"', 'My', 'dear', ',', 'since', 'I', \"'\", 've', 'chucked', 'painting', 'people', 'don', \"'\", 't', 'say', 'that', 'stuff', 'about', 'me', '--', 'they', 'say', 'it', 'about', 'Victor', 'Grindle', ',', '\"', 'was', 'his', 'only', 'protest', ',', 'as', 'he', 'rose', 'from', 'the', 'table', 'and', 'strolled', 'out', 'onto', 'the', 'sunlit', 'terrace', '.', 'I', 'glanced', 'after', 'him', ',', 'struck', 'by', 'his', 'last', 'word', '.', 'Victor', 'Grindle', 'was', ',', 'in', 'fact', ',', 'becoming', 'the', 'man', 'of', 'the', 'moment', '--', 'as', 'Jack', 'himself', ',', 'one', 'might', 'put', 'it', ',', 'had', 'been', 'the', 'man', 'of', 'the', 'hour', '.', 'The', 'younger', 'artist', 'was', 'said', 'to', 'have', 'formed', 'himself', 'at', 'my', 'friend', \"'\", 's', 'feet', ',', 'and', 'I', 'wondered', 'if', 'a', 'tinge', 'of', 'jealousy', 'underlay', 'the', 'latter', \"'\", 's', 'mysterious', 'abdication', '.', 'But', 'no', '--', 'for', 'it', 'was', 'not', 'till', 'after', 'that', 'event', 'that', 'the', '_', 'rose', 'Dubarry', '_', 'drawing-rooms', 'had', 'begun', 'to', 'display', 'their', '\"', 'Grindles', '.', '\"', 'I', 'turned', 'to', 'Mrs', '.', 'Gisburn', ',', 'who', 'had', 'lingered', 'to', 'give', 'a', 'lump', 'of', 'sugar', 'to', 'her', 'spaniel', 'in', 'the', 'dining-room', '.', '\"', 'Why', '_', 'has', '_', 'he', 'chucked', 'painting', '?', '\"', 'I', 'asked', 'abruptly', '.', 'She', 'raised', 'her', 'eyebrows', 'with', 'a', 'hint', 'of', 'good-humoured', 'surprise', '.', '\"', 'Oh', ',', 'he', 'doesn', \"'\", 't', '_', 'have', '_', 'to', 'now', ',', 'you', 'know', ';', 'and', 'I', 'want', 'him', 'to', 'enjoy', 'himself', ',', '\"', 'she', 'said', 'quite', 'simply', '.', 'I', 'looked', 'about', 'the', 'spacious', 'white-panelled', 'room', ',', 'with', 'its', '_', 'famille-verte', '_', 'vases', 'repeating', 'the', 'tones', 'of', 'the', 'pale', 'damask', 'curtains', ',', 'and', 'its', 'eighteenth-century', 'pastels', 'in', 'delicate', 'faded', 'frames', '.', '\"', 'Has', 'he', 'chucked', 'his', 'pictures', 'too', '?', 'I', 'haven', \"'\", 't', 'seen', 'a', 'single', 'one', 'in', 'the', 'house', '.', '\"', 'A', 'slight', 'shade', 'of', 'constraint', 'crossed', 'Mrs', '.', 'Gisburn', \"'\", 's', 'open', 'countenance', '.', '\"', 'It', \"'\", 's', 'his', 'ridiculous', 'modesty', ',', 'you', 'know', '.', 'He', 'says', 'they', \"'\", 're', 'not', 'fit', 'to', 'have', 'about', ';', 'he', \"'\", 's', 'sent', 'them', 'all', 'away', 'except', 'one', '--', 'my', 'portrait', '--', 'and', 'that', 'I', 'have', 'to', 'keep', 'upstairs', '.', '\"', 'His', 'ridiculous', 'modesty', '--', 'Jack', \"'\", 's', 'modesty', 'about', 'his', 'pictures', '?', 'My', 'curiosity', 'was', 'growing', 'like', 'the', 'bean-stalk', '.', 'I', 'said', 'persuasively', 'to', 'my', 'hostess', ':', '\"', 'I', 'must', 'really', 'see', 'your', 'portrait', ',', 'you', 'know', '.', '\"', 'She', 'glanced', 'out', 'almost', 'timorously', 'at', 'the', 'terrace', 'where', 'her', 'husband', ',', 'lounging', 'in', 'a', 'hooded', 'chair', ',', 'had', 'lit', 'a', 'cigar', 'and', 'drawn', 'the', 'Russian', 'deerhound', \"'\", 's', 'head', 'between', 'his', 'knees', '.', '\"', 'Well', ',', 'come', 'while', 'he', \"'\", 's', 'not', 'looking', ',', '\"', 'she', 'said', ',', 'with', 'a', 'laugh', 'that', 'tried', 'to', 'hide', 'her', 'nervousness', ';', 'and', 'I', 'followed', 'her', 'between', 'the', 'marble', 'Emperors', 'of', 'the', 'hall', ',', 'and', 'up', 'the', 'wide', 'stairs', 'with', 'terra-cotta', 'nymphs', 'poised', 'among', 'flowers', 'at', 'each', 'landing', '.', 'In', 'the', 'dimmest', 'corner', 'of', 'her', 'boudoir', ',', 'amid', 'a', 'profusion', 'of', 'delicate', 'and', 'distinguished', 'objects', ',', 'hung', 'one', 'of', 'the', 'familiar', 'oval', 'canvases', ',', 'in', 'the', 'inevitable', 'garlanded', 'frame', '.', 'The', 'mere', 'outline', 'of', 'the', 'frame', 'called', 'up', 'all', 'Gisburn', \"'\", 's', 'past', '!', 'Mrs', '.', 'Gisburn', 'drew', 'back', 'the', 'window-curtains', ',', 'moved', 'aside', 'a', '_', 'jardiniere', '_', 'full', 'of', 'pink', 'azaleas', ',', 'pushed', 'an', 'arm-chair', 'away', ',', 'and', 'said', ':', '\"', 'If', 'you', 'stand', 'here', 'you', 'can', 'just', 'manage', 'to', 'see', 'it', '.', 'I', 'had', 'it', 'over', 'the', 'mantel-piece', ',', 'but', 'he', 'wouldn', \"'\", 't', 'let', 'it', 'stay', '.', '\"', 'Yes', '--', 'I', 'could', 'just', 'manage', 'to', 'see', 'it', '--', 'the', 'first', 'portrait', 'of', 'Jack', \"'\", 's', 'I', 'had', 'ever', 'had', 'to', 'strain', 'my', 'eyes', 'over', '!', 'Usually', 'they', 'had', 'the', 'place', 'of', 'honour', '--', 'say', 'the', 'central', 'panel', 'in', 'a', 'pale', 'yellow', 'or', '_', 'rose', 'Dubarry', '_', 'drawing-room', ',', 'or', 'a', 'monumental', 'easel', 'placed', 'so', 'that', 'it', 'took', 'the', 'light', 'through', 'curtains', 'of', 'old', 'Venetian', 'point', '.', 'The', 'more', 'modest', 'place', 'became', 'the', 'picture', 'better', ';', 'yet', ',', 'as', 'my', 'eyes', 'grew', 'accustomed', 'to', 'the', 'half-light', ',', 'all', 'the', 'characteristic', 'qualities', 'came', 'out', '--', 'all', 'the', 'hesitations', 'disguised', 'as', 'audacities', ',', 'the', 'tricks', 'of', 'prestidigitation', 'by', 'which', ',', 'with', 'such', 'consummate', 'skill', ',', 'he', 'managed', 'to', 'divert', 'attention', 'from', 'the', 'real', 'business', 'of', 'the', 'picture', 'to', 'some', 'pretty', 'irrelevance', 'of', 'detail', '.', 'Mrs', '.', 'Gisburn', ',', 'presenting', 'a', 'neutral', 'surface', 'to', 'work', 'on', '--', 'forming', ',', 'as', 'it', 'were', ',', 'so', 'inevitably', 'the', 'background', 'of', 'her', 'own', 'picture', '--', 'had', 'lent', 'herself', 'in', 'an', 'unusual', 'degree', 'to', 'the', 'display', 'of', 'this', 'false', 'virtuosity', '.', 'The', 'picture', 'was', 'one', 'of', 'Jack', \"'\", 's', '\"', 'strongest', ',', '\"', 'as', 'his', 'admirers', 'would', 'have', 'put', 'it', '--', 'it', 'represented', ',', 'on', 'his', 'part', ',', 'a', 'swelling', 'of', 'muscles', ',', 'a', 'congesting', 'of', 'veins', ',', 'a', 'balancing', ',', 'straddling', 'and', 'straining', ',', 'that', 'reminded', 'one', 'of', 'the', 'circus-clown', \"'\", 's', 'ironic', 'efforts', 'to', 'lift', 'a', 'feather', '.', 'It', 'met', ',', 'in', 'short', ',', 'at', 'every', 'point', 'the', 'demand', 'of', 'lovely', 'woman', 'to', 'be', 'painted', '\"', 'strongly', '\"', 'because', 'she', 'was', 'tired', 'of', 'being', 'painted', '\"', 'sweetly', '\"', '--', 'and', 'yet', 'not', 'to', 'lose', 'an', 'atom', 'of', 'the', 'sweetness', '.', '\"', 'It', \"'\", 's', 'the', 'last', 'he', 'painted', ',', 'you', 'know', ',', '\"', 'Mrs', '.', 'Gisburn', 'said', 'with', 'pardonable', 'pride', '.', '\"', 'The', 'last', 'but', 'one', ',', '\"', 'she', 'corrected', 'herself', '--', '\"', 'but', 'the', 'other', 'doesn', \"'\", 't', 'count', ',', 'because', 'he', 'destroyed', 'it', '.', '\"', '\"', 'Destroyed', 'it', '?', '\"', 'I', 'was', 'about', 'to', 'follow', 'up', 'this', 'clue', 'when', 'I', 'heard', 'a', 'footstep', 'and', 'saw', 'Jack', 'himself', 'on', 'the', 'threshold', '.', 'As', 'he', 'stood', 'there', ',', 'his', 'hands', 'in', 'the', 'pockets', 'of', 'his', 'velveteen', 'coat', ',', 'the', 'thin', 'brown', 'waves', 'of', 'hair', 'pushed', 'back', 'from', 'his', 'white', 'forehead', ',', 'his', 'lean', 'sunburnt', 'cheeks', 'furrowed', 'by', 'a', 'smile', 'that', 'lifted', 'the', 'tips', 'of', 'a', 'self-confident', 'moustache', ',', 'I', 'felt', 'to', 'what', 'a', 'degree', 'he', 'had', 'the', 'same', 'quality', 'as', 'his', 'pictures', '--', 'the', 'quality', 'of', 'looking', 'cleverer', 'than', 'he', 'was', '.', 'His', 'wife', 'glanced', 'at', 'him', 'deprecatingly', ',', 'but', 'his', 'eyes', 'travelled', 'past', 'her', 'to', 'the', 'portrait', '.', '\"', 'Mr', '.', 'Rickham', 'wanted', 'to', 'see', 'it', ',', '\"', 'she', 'began', ',', 'as', 'if', 'excusing', 'herself', '.', 'He', 'shrugged', 'his', 'shoulders', ',', 'still', 'smiling', '.', '\"', 'Oh', ',', 'Rickham', 'found', 'me', 'out', 'long', 'ago', ',', '\"', 'he', 'said', 'lightly', ';', 'then', ',', 'passing', 'his', 'arm', 'through', 'mine', ':', '\"', 'Come', 'and', 'see', 'the', 'rest', 'of', 'the', 'house', '.', '\"', 'He', 'showed', 'it', 'to', 'me', 'with', 'a', 'kind', 'of', 'naive', 'suburban', 'pride', ':', 'the', 'bath-rooms', ',', 'the', 'speaking-tubes', ',', 'the', 'dress-closets', ',', 'the', 'trouser-presses', '--', 'all', 'the', 'complex', 'simplifications', 'of', 'the', 'millionaire', \"'\", 's', 'domestic', 'economy', '.', 'And', 'whenever', 'my', 'wonder', 'paid', 'the', 'expected', 'tribute', 'he', 'said', ',', 'throwing', 'out', 'his', 'chest', 'a', 'little', ':', '\"', 'Yes', ',', 'I', 'really', 'don', \"'\", 't', 'see', 'how', 'people', 'manage', 'to', 'live', 'without', 'that', '.', '\"', 'Well', '--', 'it', 'was', 'just', 'the', 'end', 'one', 'might', 'have', 'foreseen', 'for', 'him', '.', 'Only', 'he', 'was', ',', 'through', 'it', 'all', 'and', 'in', 'spite', 'of', 'it', 'all', '--', 'as', 'he', 'had', 'been', 'through', ',', 'and', 'in', 'spite', 'of', ',', 'his', 'pictures', '--', 'so', 'handsome', ',', 'so', 'charming', ',', 'so', 'disarming', ',', 'that', 'one', 'longed', 'to', 'cry', 'out', ':', '\"', 'Be', 'dissatisfied', 'with', 'your', 'leisure', '!', '\"', 'as', 'once', 'one', 'had', 'longed', 'to', 'say', ':', '\"', 'Be', 'dissatisfied', 'with', 'your', 'work', '!', '\"', 'But', ',', 'with', 'the', 'cry', 'on', 'my', 'lips', ',', 'my', 'diagnosis', 'suffered', 'an', 'unexpected', 'check', '.', '\"', 'This', 'is', 'my', 'own', 'lair', ',', '\"', 'he', 'said', ',', 'leading', 'me', 'into', 'a', 'dark', 'plain', 'room', 'at', 'the', 'end', 'of', 'the', 'florid', 'vista', '.', 'It', 'was', 'square', 'and', 'brown', 'and', 'leathery', ':', 'no', '\"', 'effects', '\"', ';', 'no', 'bric-a-brac', ',', 'none', 'of', 'the', 'air', 'of', 'posing', 'for', 'reproduction', 'in', 'a', 'picture', 'weekly', '--', 'above', 'all', ',', 'no', 'least', 'sign', 'of', 'ever', 'having', 'been', 'used', 'as', 'a', 'studio', '.', 'The', 'fact', 'brought', 'home', 'to', 'me', 'the', 'absolute', 'finality', 'of', 'Jack', \"'\", 's', 'break', 'with', 'his', 'old', 'life', '.', '\"', 'Don', \"'\", 't', 'you', 'ever', 'dabble', 'with', 'paint', 'any', 'more', '?', '\"', 'I', 'asked', ',', 'still', 'looking', 'about', 'for', 'a', 'trace', 'of', 'such', 'activity', '.', '\"', 'Never', ',', '\"', 'he', 'said', 'briefly', '.', '\"', 'Or', 'water-colour', '--', 'or', 'etching', '?', '\"', 'His', 'confident', 'eyes', 'grew', 'dim', ',', 'and', 'his', 'cheeks', 'paled', 'a', 'little', 'under', 'their', 'handsome', 'sunburn', '.', '\"', 'Never', 'think', 'of', 'it', ',', 'my', 'dear', 'fellow', '--', 'any', 'more', 'than', 'if', 'I', \"'\", 'd', 'never', 'touched', 'a', 'brush', '.', '\"', 'And', 'his', 'tone', 'told', 'me', 'in', 'a', 'flash', 'that', 'he', 'never', 'thought', 'of', 'anything', 'else', '.', 'I', 'moved', 'away', ',', 'instinctively', 'embarrassed', 'by', 'my', 'unexpected', 'discovery', ';', 'and', 'as', 'I', 'turned', ',', 'my', 'eye', 'fell', 'on', 'a', 'small', 'picture', 'above', 'the', 'mantel-piece', '--', 'the', 'only', 'object', 'breaking', 'the', 'plain', 'oak', 'panelling', 'of', 'the', 'room', '.', '\"', 'Oh', ',', 'by', 'Jove', '!', '\"', 'I', 'said', '.', 'It', 'was', 'a', 'sketch', 'of', 'a', 'donkey', '--', 'an', 'old', 'tired', 'donkey', ',', 'standing', 'in', 'the', 'rain', 'under', 'a', 'wall', '.', '\"', 'By', 'Jove', '--', 'a', 'Stroud', '!', '\"', 'I', 'cried', '.', 'He', 'was', 'silent', ';', 'but', 'I', 'felt', 'him', 'close', 'behind', 'me', ',', 'breathing', 'a', 'little', 'quickly', '.', '\"', 'What', 'a', 'wonder', '!', 'Made', 'with', 'a', 'dozen', 'lines', '--', 'but', 'on', 'everlasting', 'foundations', '.', 'You', 'lucky', 'chap', ',', 'where', 'did', 'you', 'get', 'it', '?', '\"', 'He', 'answered', 'slowly', ':', '\"', 'Mrs', '.', 'Stroud', 'gave', 'it', 'to', 'me', '.', '\"', '\"', 'Ah', '--', 'I', 'didn', \"'\", 't', 'know', 'you', 'even', 'knew', 'the', 'Strouds', '.', 'He', 'was', 'such', 'an', 'inflexible', 'hermit', '.', '\"', '\"', 'I', 'didn', \"'\", 't', '--', 'till', 'after', '.', '.', '.', '.', 'She', 'sent', 'for', 'me', 'to', 'paint', 'him', 'when', 'he', 'was', 'dead', '.', '\"', '\"', 'When', 'he', 'was', 'dead', '?', 'You', '?', '\"', 'I', 'must', 'have', 'let', 'a', 'little', 'too', 'much', 'amazement', 'escape', 'through', 'my', 'surprise', ',', 'for', 'he', 'answered', 'with', 'a', 'deprecating', 'laugh', ':', '\"', 'Yes', '--', 'she', \"'\", 's', 'an', 'awful', 'simpleton', ',', 'you', 'know', ',', 'Mrs', '.', 'Stroud', '.', 'Her', 'only', 'idea', 'was', 'to', 'have', 'him', 'done', 'by', 'a', 'fashionable', 'painter', '--', 'ah', ',', 'poor', 'Stroud', '!', 'She', 'thought', 'it', 'the', 'surest', 'way', 'of', 'proclaiming', 'his', 'greatness', '--', 'of', 'forcing', 'it', 'on', 'a', 'purblind', 'public', '.', 'And', 'at', 'the', 'moment', 'I', 'was', '_', 'the', '_', 'fashionable', 'painter', '.', '\"', '\"', 'Ah', ',', 'poor', 'Stroud', '--', 'as', 'you', 'say', '.', 'Was', '_', 'that', '_', 'his', 'history', '?', '\"', '\"', 'That', 'was', 'his', 'history', '.', 'She', 'believed', 'in', 'him', ',', 'gloried', 'in', 'him', '--', 'or', 'thought', 'she', 'did', '.', 'But', 'she', 'couldn', \"'\", 't', 'bear', 'not', 'to', 'have', 'all', 'the', 'drawing-rooms', 'with', 'her', '.', 'She', 'couldn', \"'\", 't', 'bear', 'the', 'fact', 'that', ',', 'on', 'varnishing', 'days', ',', 'one', 'could', 'always', 'get', 'near', 'enough', 'to', 'see', 'his', 'pictures', '.', 'Poor', 'woman', '!', 'She', \"'\", 's', 'just', 'a', 'fragment', 'groping', 'for', 'other', 'fragments', '.', 'Stroud', 'is', 'the', 'only', 'whole', 'I', 'ever', 'knew', '.', '\"', '\"', 'You', 'ever', 'knew', '?', 'But', 'you', 'just', 'said', '--', '\"', 'Gisburn', 'had', 'a', 'curious', 'smile', 'in', 'his', 'eyes', '.', '\"', 'Oh', ',', 'I', 'knew', 'him', ',', 'and', 'he', 'knew', 'me', '--', 'only', 'it', 'happened', 'after', 'he', 'was', 'dead', '.', '\"', 'I', 'dropped', 'my', 'voice', 'instinctively', '.', '\"', 'When', 'she', 'sent', 'for', 'you', '?', '\"', '\"', 'Yes', '--', 'quite', 'insensible', 'to', 'the', 'irony', '.', 'She', 'wanted', 'him', 'vindicated', '--', 'and', 'by', 'me', '!', '\"', 'He', 'laughed', 'again', ',', 'and', 'threw', 'back', 'his', 'head', 'to', 'look', 'up', 'at', 'the', 'sketch', 'of', 'the', 'donkey', '.', '\"', 'There', 'were', 'days', 'when', 'I', 'couldn', \"'\", 't', 'look', 'at', 'that', 'thing', '--', 'couldn', \"'\", 't', 'face', 'it', '.', 'But', 'I', 'forced', 'myself', 'to', 'put', 'it', 'here', ';', 'and', 'now', 'it', \"'\", 's', 'cured', 'me', '--', 'cured', 'me', '.', 'That', \"'\", 's', 'the', 'reason', 'why', 'I', 'don', \"'\", 't', 'dabble', 'any', 'more', ',', 'my', 'dear', 'Rickham', ';', 'or', 'rather', 'Stroud', 'himself', 'is', 'the', 'reason', '.', '\"', 'For', 'the', 'first', 'time', 'my', 'idle', 'curiosity', 'about', 'my', 'companion', 'turned', 'into', 'a', 'serious', 'desire', 'to', 'understand', 'him', 'better', '.', '\"', 'I', 'wish', 'you', \"'\", 'd', 'tell', 'me', 'how', 'it', 'happened', ',', '\"', 'I', 'said', '.', 'He', 'stood', 'looking', 'up', 'at', 'the', 'sketch', ',', 'and', 'twirling', 'between', 'his', 'fingers', 'a', 'cigarette', 'he', 'had', 'forgotten', 'to', 'light', '.', 'Suddenly', 'he', 'turned', 'toward', 'me', '.', '\"', 'I', \"'\", 'd', 'rather', 'like', 'to', 'tell', 'you', '--', 'because', 'I', \"'\", 've', 'always', 'suspected', 'you', 'of', 'loathing', 'my', 'work', '.', '\"', 'I', 'made', 'a', 'deprecating', 'gesture', ',', 'which', 'he', 'negatived', 'with', 'a', 'good-humoured', 'shrug', '.', '\"', 'Oh', ',', 'I', 'didn', \"'\", 't', 'care', 'a', 'straw', 'when', 'I', 'believed', 'in', 'myself', '--', 'and', 'now', 'it', \"'\", 's', 'an', 'added', 'tie', 'between', 'us', '!', '\"', 'He', 'laughed', 'slightly', ',', 'without', 'bitterness', ',', 'and', 'pushed', 'one', 'of', 'the', 'deep', 'arm-chairs', 'forward', '.', '\"', 'There', ':', 'make', 'yourself', 'comfortable', '--', 'and', 'here', 'are', 'the', 'cigars', 'you', 'like', '.', '\"', 'He', 'placed', 'them', 'at', 'my', 'elbow', 'and', 'continued', 'to', 'wander', 'up', 'and', 'down', 'the', 'room', ',', 'stopping', 'now', 'and', 'then', 'beneath', 'the', 'picture', '.', '\"', 'How', 'it', 'happened', '?', 'I', 'can', 'tell', 'you', 'in', 'five', 'minutes', '--', 'and', 'it', 'didn', \"'\", 't', 'take', 'much', 'longer', 'to', 'happen', '.', '.', '.', '.', 'I', 'can', 'remember', 'now', 'how', 'surprised', 'and', 'pleased', 'I', 'was', 'when', 'I', 'got', 'Mrs', '.', 'Stroud', \"'\", 's', 'note', '.', 'Of', 'course', ',', 'deep', 'down', ',', 'I', 'had', 'always', '_', 'felt', '_', 'there', 'was', 'no', 'one', 'like', 'him', '--', 'only', 'I', 'had', 'gone', 'with', 'the', 'stream', ',', 'echoed', 'the', 'usual', 'platitudes', 'about', 'him', ',', 'till', 'I', 'half', 'got', 'to', 'think', 'he', 'was', 'a', 'failure', ',', 'one', 'of', 'the', 'kind', 'that', 'are', 'left', 'behind', '.', 'By', 'Jove', ',', 'and', 'he', '_', 'was', '_', 'left', 'behind', '--', 'because', 'he', 'had', 'come', 'to', 'stay', '!', 'The', 'rest', 'of', 'us', 'had', 'to', 'let', 'ourselves', 'be', 'swept', 'along', 'or', 'go', 'under', ',', 'but', 'he', 'was', 'high', 'above', 'the', 'current', '--', 'on', 'everlasting', 'foundations', ',', 'as', 'you', 'say', '.', '\"', 'Well', ',', 'I', 'went', 'off', 'to', 'the', 'house', 'in', 'my', 'most', 'egregious', 'mood', '--', 'rather', 'moved', ',', 'Lord', 'forgive', 'me', ',', 'at', 'the', 'pathos', 'of', 'poor', 'Stroud', \"'\", 's', 'career', 'of', 'failure', 'being', 'crowned', 'by', 'the', 'glory', 'of', 'my', 'painting', 'him', '!', 'Of', 'course', 'I', 'meant', 'to', 'do', 'the', 'picture', 'for', 'nothing', '--', 'I', 'told', 'Mrs', '.', 'Stroud', 'so', 'when', 'she', 'began', 'to', 'stammer', 'something', 'about', 'her', 'poverty', '.', 'I', 'remember', 'getting', 'off', 'a', 'prodigious', 'phrase', 'about', 'the', 'honour', 'being', '_', 'mine', '_', '--', 'oh', ',', 'I', 'was', 'princely', ',', 'my', 'dear', 'Rickham', '!', 'I', 'was', 'posing', 'to', 'myself', 'like', 'one', 'of', 'my', 'own', 'sitters', '.', '\"', 'Then', 'I', 'was', 'taken', 'up', 'and', 'left', 'alone', 'with', 'him', '.', 'I', 'had', 'sent', 'all', 'my', 'traps', 'in', 'advance', ',', 'and', 'I', 'had', 'only', 'to', 'set', 'up', 'the', 'easel', 'and', 'get', 'to', 'work', '.', 'He', 'had', 'been', 'dead', 'only', 'twenty-four', 'hours', ',', 'and', 'he', 'died', 'suddenly', ',', 'of', 'heart', 'disease', ',', 'so', 'that', 'there', 'had', 'been', 'no', 'preliminary', 'work', 'of', 'destruction', '--', 'his', 'face', 'was', 'clear', 'and', 'untouched', '.', 'I', 'had', 'met', 'him', 'once', 'or', 'twice', ',', 'years', 'before', ',', 'and', 'thought', 'him', 'insignificant', 'and', 'dingy', '.', 'Now', 'I', 'saw', 'that', 'he', 'was', 'superb', '.', '\"', 'I', 'was', 'glad', 'at', 'first', ',', 'with', 'a', 'merely', 'aesthetic', 'satisfaction', ':', 'glad', 'to', 'have', 'my', 'hand', 'on', 'such', 'a', \"'\", 'subject', '.', \"'\", 'Then', 'his', 'strange', 'life-likeness', 'began', 'to', 'affect', 'me', 'queerly', '--', 'as', 'I', 'blocked', 'the', 'head', 'in', 'I', 'felt', 'as', 'if', 'he', 'were', 'watching', 'me', 'do', 'it', '.', 'The', 'sensation', 'was', 'followed', 'by', 'the', 'thought', ':', 'if', 'he', '_', 'were', '_', 'watching', 'me', ',', 'what', 'would', 'he', 'say', 'to', 'my', 'way', 'of', 'working', '?', 'My', 'strokes', 'began', 'to', 'go', 'a', 'little', 'wild', '--', 'I', 'felt', 'nervous', 'and', 'uncertain', '.', '\"', 'Once', ',', 'when', 'I', 'looked', 'up', ',', 'I', 'seemed', 'to', 'see', 'a', 'smile', 'behind', 'his', 'close', 'grayish', 'beard', '--', 'as', 'if', 'he', 'had', 'the', 'secret', ',', 'and', 'were', 'amusing', 'himself', 'by', 'holding', 'it', 'back', 'from', 'me', '.', 'That', 'exasperated', 'me', 'still', 'more', '.', 'The', 'secret', '?', 'Why', ',', 'I', 'had', 'a', 'secret', 'worth', 'twenty', 'of', 'his', '!', 'I', 'dashed', 'at', 'the', 'canvas', 'furiously', ',', 'and', 'tried', 'some', 'of', 'my', 'bravura', 'tricks', '.', 'But', 'they', 'failed', 'me', ',', 'they', 'crumbled', '.', 'I', 'saw', 'that', 'he', 'wasn', \"'\", 't', 'watching', 'the', 'showy', 'bits', '--', 'I', 'couldn', \"'\", 't', 'distract', 'his', 'attention', ';', 'he', 'just', 'kept', 'his', 'eyes', 'on', 'the', 'hard', 'passages', 'between', '.', 'Those', 'were', 'the', 'ones', 'I', 'had', 'always', 'shirked', ',', 'or', 'covered', 'up', 'with', 'some', 'lying', 'paint', '.', 'And', 'how', 'he', 'saw', 'through', 'my', 'lies', '!', '\"', 'I', 'looked', 'up', 'again', ',', 'and', 'caught', 'sight', 'of', 'that', 'sketch', 'of', 'the', 'donkey', 'hanging', 'on', 'the', 'wall', 'near', 'his', 'bed', '.', 'His', 'wife', 'told', 'me', 'afterward', 'it', 'was', 'the', 'last', 'thing', 'he', 'had', 'done', '--', 'just', 'a', 'note', 'taken', 'with', 'a', 'shaking', 'hand', ',', 'when', 'he', 'was', 'down', 'in', 'Devonshire', 'recovering', 'from', 'a', 'previous', 'heart', 'attack', '.', 'Just', 'a', 'note', '!', 'But', 'it', 'tells', 'his', 'whole', 'history', '.', 'There', 'are', 'years', 'of', 'patient', 'scornful', 'persistence', 'in', 'every', 'line', '.', 'A', 'man', 'who', 'had', 'swum', 'with', 'the', 'current', 'could', 'never', 'have', 'learned', 'that', 'mighty', 'up-stream', 'stroke', '.', '.', '.', '.', '\"', 'I', 'turned', 'back', 'to', 'my', 'work', ',', 'and', 'went', 'on', 'groping', 'and', 'muddling', ';', 'then', 'I', 'looked', 'at', 'the', 'donkey', 'again', '.', 'I', 'saw', 'that', ',', 'when', 'Stroud', 'laid', 'in', 'the', 'first', 'stroke', ',', 'he', 'knew', 'just', 'what', 'the', 'end', 'would', 'be', '.', 'He', 'had', 'possessed', 'his', 'subject', ',', 'absorbed', 'it', ',', 'recreated', 'it', '.', 'When', 'had', 'I', 'done', 'that', 'with', 'any', 'of', 'my', 'things', '?', 'They', 'hadn', \"'\", 't', 'been', 'born', 'of', 'me', '--', 'I', 'had', 'just', 'adopted', 'them', '.', '.', '.', '.', '\"', 'Hang', 'it', ',', 'Rickham', ',', 'with', 'that', 'face', 'watching', 'me', 'I', 'couldn', \"'\", 't', 'do', 'another', 'stroke', '.', 'The', 'plain', 'truth', 'was', ',', 'I', 'didn', \"'\", 't', 'know', 'where', 'to', 'put', 'it', '--', '_', 'I', 'had', 'never', 'known', '_', '.', 'Only', ',', 'with', 'my', 'sitters', 'and', 'my', 'public', ',', 'a', 'showy', 'splash', 'of', 'colour', 'covered', 'up', 'the', 'fact', '--', 'I', 'just', 'threw', 'paint', 'into', 'their', 'faces', '.', '.', '.', '.', 'Well', ',', 'paint', 'was', 'the', 'one', 'medium', 'those', 'dead', 'eyes', 'could', 'see', 'through', '--', 'see', 'straight', 'to', 'the', 'tottering', 'foundations', 'underneath', '.', 'Don', \"'\", 't', 'you', 'know', 'how', ',', 'in', 'talking', 'a', 'foreign', 'language', ',', 'even', 'fluently', ',', 'one', 'says', 'half', 'the', 'time', 'not', 'what', 'one', 'wants', 'to', 'but', 'what', 'one', 'can', '?', 'Well', '--', 'that', 'was', 'the', 'way', 'I', 'painted', ';', 'and', 'as', 'he', 'lay', 'there', 'and', 'watched', 'me', ',', 'the', 'thing', 'they', 'called', 'my', \"'\", 'technique', \"'\", 'collapsed', 'like', 'a', 'house', 'of', 'cards', '.', 'He', 'didn', \"'\", 't', 'sneer', ',', 'you', 'understand', ',', 'poor', 'Stroud', '--', 'he', 'just', 'lay', 'there', 'quietly', 'watching', ',', 'and', 'on', 'his', 'lips', ',', 'through', 'the', 'gray', 'beard', ',', 'I', 'seemed', 'to', 'hear', 'the', 'question', ':', \"'\", 'Are', 'you', 'sure', 'you', 'know', 'where', 'you', \"'\", 're', 'coming', 'out', '?', \"'\", '\"', 'If', 'I', 'could', 'have', 'painted', 'that', 'face', ',', 'with', 'that', 'question', 'on', 'it', ',', 'I', 'should', 'have', 'done', 'a', 'great', 'thing', '.', 'The', 'next', 'greatest', 'thing', 'was', 'to', 'see', 'that', 'I', 'couldn', \"'\", 't', '--', 'and', 'that', 'grace', 'was', 'given', 'me', '.', 'But', ',', 'oh', ',', 'at', 'that', 'minute', ',', 'Rickham', ',', 'was', 'there', 'anything', 'on', 'earth', 'I', 'wouldn', \"'\", 't', 'have', 'given', 'to', 'have', 'Stroud', 'alive', 'before', 'me', ',', 'and', 'to', 'hear', 'him', 'say', ':', \"'\", 'It', \"'\", 's', 'not', 'too', 'late', '--', 'I', \"'\", 'll', 'show', 'you', 'how', \"'\", '?', '\"', 'It', '_', 'was', '_', 'too', 'late', '--', 'it', 'would', 'have', 'been', ',', 'even', 'if', 'he', \"'\", 'd', 'been', 'alive', '.', 'I', 'packed', 'up', 'my', 'traps', ',', 'and', 'went', 'down', 'and', 'told', 'Mrs', '.', 'Stroud', '.', 'Of', 'course', 'I', 'didn', \"'\", 't', 'tell', 'her', '_', 'that', '_', '--', 'it', 'would', 'have', 'been', 'Greek', 'to', 'her', '.', 'I', 'simply', 'said', 'I', 'couldn', \"'\", 't', 'paint', 'him', ',', 'that', 'I', 'was', 'too', 'moved', '.', 'She', 'rather', 'liked', 'the', 'idea', '--', 'she', \"'\", 's', 'so', 'romantic', '!', 'It', 'was', 'that', 'that', 'made', 'her', 'give', 'me', 'the', 'donkey', '.', 'But', 'she', 'was', 'terribly', 'upset', 'at', 'not', 'getting', 'the', 'portrait', '--', 'she', 'did', 'so', 'want', 'him', \"'\", 'done', \"'\", 'by', 'some', 'one', 'showy', '!', 'At', 'first', 'I', 'was', 'afraid', 'she', 'wouldn', \"'\", 't', 'let', 'me', 'off', '--', 'and', 'at', 'my', 'wits', \"'\", 'end', 'I', 'suggested', 'Grindle', '.', 'Yes', ',', 'it', 'was', 'I', 'who', 'started', 'Grindle', ':', 'I', 'told', 'Mrs', '.', 'Stroud', 'he', 'was', 'the', \"'\", 'coming', \"'\", 'man', ',', 'and', 'she', 'told', 'somebody', 'else', ',', 'and', 'so', 'it', 'got', 'to', 'be', 'true', '.', '.', '.', '.', 'And', 'he', 'painted', 'Stroud', 'without', 'wincing', ';', 'and', 'she', 'hung', 'the', 'picture', 'among', 'her', 'husband', \"'\", 's', 'things', '.', '.', '.', '.', '\"', 'He', 'flung', 'himself', 'down', 'in', 'the', 'arm-chair', 'near', 'mine', ',', 'laid', 'back', 'his', 'head', ',', 'and', 'clasping', 'his', 'arms', 'beneath', 'it', ',', 'looked', 'up', 'at', 'the', 'picture', 'above', 'the', 'chimney-piece', '.', '\"', 'I', 'like', 'to', 'fancy', 'that', 'Stroud', 'himself', 'would', 'have', 'given', 'it', 'to', 'me', ',', 'if', 'he', \"'\", 'd', 'been', 'able', 'to', 'say', 'what', 'he', 'thought', 'that', 'day', '.', '\"', 'And', ',', 'in', 'answer', 'to', 'a', 'question', 'I', 'put', 'half-mechanically', '--', '\"', 'Begin', 'again', '?', '\"', 'he', 'flashed', 'out', '.', '\"', 'When', 'the', 'one', 'thing', 'that', 'brings', 'me', 'anywhere', 'near', 'him', 'is', 'that', 'I', 'knew', 'enough', 'to', 'leave', 'off', '?', '\"', 'He', 'stood', 'up', 'and', 'laid', 'his', 'hand', 'on', 'my', 'shoulder', 'with', 'a', 'laugh', '.', '\"', 'Only', 'the', 'irony', 'of', 'it', 'is', 'that', 'I', '_', 'am', '_', 'still', 'painting', '--', 'since', 'Grindle', \"'\", 's', 'doing', 'it', 'for', 'me', '!', 'The', 'Strouds', 'stand', 'alone', ',', 'and', 'happen', 'once', '--', 'but', 'there', \"'\", 's', 'no', 'exterminating', 'our', 'kind', 'of', 'art', '.', '\"']\n"
     ]
    }
   ],
   "source": [
    "print(processed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "18ec4f4a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4690"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(processed)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbaf013d",
   "metadata": {},
   "source": [
    "## 将token转化为token ID（标识符）\n",
    "\n",
    "- ​接下来，我们将文本token转化为token ID（标识符），以便后续通过嵌入层（embedding layers）进行处理。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0054b81e",
   "metadata": {},
   "source": [
    "![](./images/tokenizer2id.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83e17779",
   "metadata": {},
   "source": [
    "- 基于这些token，我们可以构建一个包含所有唯一token的词汇表（vocabulary）"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f0ffd94",
   "metadata": {},
   "source": [
    "- ​下文通过一个小规模词汇表（small vocabulary），演示对短样本文本的分词（tokenization）过程："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b0f71894",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1130"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_words = sorted(set(processed))\n",
    "\n",
    "len(all_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "458f96b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = {token: integer for integer, token in enumerate(all_words)}\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d67ffd26",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'!': 0,\n",
       " '\"': 1,\n",
       " \"'\": 2,\n",
       " '(': 3,\n",
       " ')': 4,\n",
       " ',': 5,\n",
       " '--': 6,\n",
       " '.': 7,\n",
       " ':': 8,\n",
       " ';': 9,\n",
       " '?': 10,\n",
       " 'A': 11,\n",
       " 'Ah': 12,\n",
       " 'Among': 13,\n",
       " 'And': 14,\n",
       " 'Are': 15,\n",
       " 'Arrt': 16,\n",
       " 'As': 17,\n",
       " 'At': 18,\n",
       " 'Be': 19,\n",
       " 'Begin': 20,\n",
       " 'Burlington': 21,\n",
       " 'But': 22,\n",
       " 'By': 23,\n",
       " 'Carlo': 24,\n",
       " 'Chicago': 25,\n",
       " 'Claude': 26,\n",
       " 'Come': 27,\n",
       " 'Croft': 28,\n",
       " 'Destroyed': 29,\n",
       " 'Devonshire': 30,\n",
       " 'Don': 31,\n",
       " 'Dubarry': 32,\n",
       " 'Emperors': 33,\n",
       " 'Florence': 34,\n",
       " 'For': 35,\n",
       " 'Gallery': 36,\n",
       " 'Gideon': 37,\n",
       " 'Gisburn': 38,\n",
       " 'Gisburns': 39,\n",
       " 'Grafton': 40,\n",
       " 'Greek': 41,\n",
       " 'Grindle': 42,\n",
       " 'Grindles': 43,\n",
       " 'HAD': 44,\n",
       " 'Had': 45,\n",
       " 'Hang': 46,\n",
       " 'Has': 47,\n",
       " 'He': 48,\n",
       " 'Her': 49,\n",
       " 'Hermia': 50,\n",
       " 'His': 51,\n",
       " 'How': 52,\n",
       " 'I': 53,\n",
       " 'If': 54,\n",
       " 'In': 55,\n",
       " 'It': 56,\n",
       " 'Jack': 57,\n",
       " 'Jove': 58,\n",
       " 'Just': 59,\n",
       " 'Lord': 60,\n",
       " 'Made': 61,\n",
       " 'Miss': 62,\n",
       " 'Money': 63,\n",
       " 'Monte': 64,\n",
       " 'Moon-dancers': 65,\n",
       " 'Mr': 66,\n",
       " 'Mrs': 67,\n",
       " 'My': 68,\n",
       " 'Never': 69,\n",
       " 'No': 70,\n",
       " 'Now': 71,\n",
       " 'Nutley': 72,\n",
       " 'Of': 73,\n",
       " 'Oh': 74,\n",
       " 'On': 75,\n",
       " 'Once': 76,\n",
       " 'Only': 77,\n",
       " 'Or': 78,\n",
       " 'Perhaps': 79,\n",
       " 'Poor': 80,\n",
       " 'Professional': 81,\n",
       " 'Renaissance': 82,\n",
       " 'Rickham': 83,\n",
       " 'Riviera': 84,\n",
       " 'Rome': 85,\n",
       " 'Russian': 86,\n",
       " 'Sevres': 87,\n",
       " 'She': 88,\n",
       " 'Stroud': 89,\n",
       " 'Strouds': 90,\n",
       " 'Suddenly': 91,\n",
       " 'That': 92,\n",
       " 'The': 93,\n",
       " 'Then': 94,\n",
       " 'There': 95,\n",
       " 'They': 96,\n",
       " 'This': 97,\n",
       " 'Those': 98,\n",
       " 'Though': 99,\n",
       " 'Thwing': 100,\n",
       " 'Thwings': 101,\n",
       " 'To': 102,\n",
       " 'Usually': 103,\n",
       " 'Venetian': 104,\n",
       " 'Victor': 105,\n",
       " 'Was': 106,\n",
       " 'We': 107,\n",
       " 'Well': 108,\n",
       " 'What': 109,\n",
       " 'When': 110,\n",
       " 'Why': 111,\n",
       " 'Yes': 112,\n",
       " 'You': 113,\n",
       " '_': 114,\n",
       " 'a': 115,\n",
       " 'abdication': 116,\n",
       " 'able': 117,\n",
       " 'about': 118,\n",
       " 'above': 119,\n",
       " 'abruptly': 120,\n",
       " 'absolute': 121,\n",
       " 'absorbed': 122,\n",
       " 'absurdity': 123,\n",
       " 'academic': 124,\n",
       " 'accuse': 125,\n",
       " 'accustomed': 126,\n",
       " 'across': 127,\n",
       " 'activity': 128,\n",
       " 'add': 129,\n",
       " 'added': 130,\n",
       " 'admirers': 131,\n",
       " 'adopted': 132,\n",
       " 'adulation': 133,\n",
       " 'advance': 134,\n",
       " 'aesthetic': 135,\n",
       " 'affect': 136,\n",
       " 'afraid': 137,\n",
       " 'after': 138,\n",
       " 'afterward': 139,\n",
       " 'again': 140,\n",
       " 'ago': 141,\n",
       " 'ah': 142,\n",
       " 'air': 143,\n",
       " 'alive': 144,\n",
       " 'all': 145,\n",
       " 'almost': 146,\n",
       " 'alone': 147,\n",
       " 'along': 148,\n",
       " 'always': 149,\n",
       " 'am': 150,\n",
       " 'amazement': 151,\n",
       " 'amid': 152,\n",
       " 'among': 153,\n",
       " 'amplest': 154,\n",
       " 'amusing': 155,\n",
       " 'an': 156,\n",
       " 'and': 157,\n",
       " 'another': 158,\n",
       " 'answer': 159,\n",
       " 'answered': 160,\n",
       " 'any': 161,\n",
       " 'anything': 162,\n",
       " 'anywhere': 163,\n",
       " 'apparent': 164,\n",
       " 'apparently': 165,\n",
       " 'appearance': 166,\n",
       " 'appeared': 167,\n",
       " 'appointed': 168,\n",
       " 'are': 169,\n",
       " 'arm': 170,\n",
       " 'arm-chair': 171,\n",
       " 'arm-chairs': 172,\n",
       " 'arms': 173,\n",
       " 'art': 174,\n",
       " 'articles': 175,\n",
       " 'artist': 176,\n",
       " 'as': 177,\n",
       " 'aside': 178,\n",
       " 'asked': 179,\n",
       " 'at': 180,\n",
       " 'atmosphere': 181,\n",
       " 'atom': 182,\n",
       " 'attack': 183,\n",
       " 'attention': 184,\n",
       " 'attitude': 185,\n",
       " 'audacities': 186,\n",
       " 'away': 187,\n",
       " 'awful': 188,\n",
       " 'axioms': 189,\n",
       " 'azaleas': 190,\n",
       " 'back': 191,\n",
       " 'background': 192,\n",
       " 'balance': 193,\n",
       " 'balancing': 194,\n",
       " 'balustraded': 195,\n",
       " 'basking': 196,\n",
       " 'bath-rooms': 197,\n",
       " 'be': 198,\n",
       " 'beaming': 199,\n",
       " 'bean-stalk': 200,\n",
       " 'bear': 201,\n",
       " 'beard': 202,\n",
       " 'beauty': 203,\n",
       " 'became': 204,\n",
       " 'because': 205,\n",
       " 'becoming': 206,\n",
       " 'bed': 207,\n",
       " 'been': 208,\n",
       " 'before': 209,\n",
       " 'began': 210,\n",
       " 'begun': 211,\n",
       " 'behind': 212,\n",
       " 'being': 213,\n",
       " 'believed': 214,\n",
       " 'beneath': 215,\n",
       " 'bespoke': 216,\n",
       " 'better': 217,\n",
       " 'between': 218,\n",
       " 'big': 219,\n",
       " 'bits': 220,\n",
       " 'bitterness': 221,\n",
       " 'blocked': 222,\n",
       " 'born': 223,\n",
       " 'borne': 224,\n",
       " 'boudoir': 225,\n",
       " 'bravura': 226,\n",
       " 'break': 227,\n",
       " 'breaking': 228,\n",
       " 'breathing': 229,\n",
       " 'bric-a-brac': 230,\n",
       " 'briefly': 231,\n",
       " 'brings': 232,\n",
       " 'bronzes': 233,\n",
       " 'brought': 234,\n",
       " 'brown': 235,\n",
       " 'brush': 236,\n",
       " 'bull': 237,\n",
       " 'business': 238,\n",
       " 'but': 239,\n",
       " 'buying': 240,\n",
       " 'by': 241,\n",
       " 'called': 242,\n",
       " 'came': 243,\n",
       " 'can': 244,\n",
       " 'canvas': 245,\n",
       " 'canvases': 246,\n",
       " 'cards': 247,\n",
       " 'care': 248,\n",
       " 'career': 249,\n",
       " 'caught': 250,\n",
       " 'central': 251,\n",
       " 'chair': 252,\n",
       " 'chap': 253,\n",
       " 'characteristic': 254,\n",
       " 'charming': 255,\n",
       " 'cheap': 256,\n",
       " 'check': 257,\n",
       " 'cheeks': 258,\n",
       " 'chest': 259,\n",
       " 'chimney-piece': 260,\n",
       " 'chucked': 261,\n",
       " 'cigar': 262,\n",
       " 'cigarette': 263,\n",
       " 'cigars': 264,\n",
       " 'circulation': 265,\n",
       " 'circumstance': 266,\n",
       " 'circus-clown': 267,\n",
       " 'claimed': 268,\n",
       " 'clasping': 269,\n",
       " 'clear': 270,\n",
       " 'cleverer': 271,\n",
       " 'close': 272,\n",
       " 'clue': 273,\n",
       " 'coat': 274,\n",
       " 'collapsed': 275,\n",
       " 'colour': 276,\n",
       " 'come': 277,\n",
       " 'comfortable': 278,\n",
       " 'coming': 279,\n",
       " 'companion': 280,\n",
       " 'compared': 281,\n",
       " 'complex': 282,\n",
       " 'confident': 283,\n",
       " 'congesting': 284,\n",
       " 'conjugal': 285,\n",
       " 'constraint': 286,\n",
       " 'consummate': 287,\n",
       " 'contended': 288,\n",
       " 'continued': 289,\n",
       " 'corner': 290,\n",
       " 'corrected': 291,\n",
       " 'could': 292,\n",
       " 'couldn': 293,\n",
       " 'count': 294,\n",
       " 'countenance': 295,\n",
       " 'couple': 296,\n",
       " 'course': 297,\n",
       " 'covered': 298,\n",
       " 'craft': 299,\n",
       " 'cried': 300,\n",
       " 'crossed': 301,\n",
       " 'crowned': 302,\n",
       " 'crumbled': 303,\n",
       " 'cry': 304,\n",
       " 'cured': 305,\n",
       " 'curiosity': 306,\n",
       " 'curious': 307,\n",
       " 'current': 308,\n",
       " 'curtains': 309,\n",
       " 'd': 310,\n",
       " 'dabble': 311,\n",
       " 'damask': 312,\n",
       " 'dark': 313,\n",
       " 'dashed': 314,\n",
       " 'day': 315,\n",
       " 'days': 316,\n",
       " 'dead': 317,\n",
       " 'deadening': 318,\n",
       " 'dear': 319,\n",
       " 'deep': 320,\n",
       " 'deerhound': 321,\n",
       " 'degree': 322,\n",
       " 'delicate': 323,\n",
       " 'demand': 324,\n",
       " 'denied': 325,\n",
       " 'deploring': 326,\n",
       " 'deprecating': 327,\n",
       " 'deprecatingly': 328,\n",
       " 'desire': 329,\n",
       " 'destroyed': 330,\n",
       " 'destruction': 331,\n",
       " 'desultory': 332,\n",
       " 'detail': 333,\n",
       " 'diagnosis': 334,\n",
       " 'did': 335,\n",
       " 'didn': 336,\n",
       " 'died': 337,\n",
       " 'dim': 338,\n",
       " 'dimmest': 339,\n",
       " 'dingy': 340,\n",
       " 'dining-room': 341,\n",
       " 'disarming': 342,\n",
       " 'discovery': 343,\n",
       " 'discrimination': 344,\n",
       " 'discussion': 345,\n",
       " 'disdain': 346,\n",
       " 'disdained': 347,\n",
       " 'disease': 348,\n",
       " 'disguised': 349,\n",
       " 'display': 350,\n",
       " 'dissatisfied': 351,\n",
       " 'distinguished': 352,\n",
       " 'distract': 353,\n",
       " 'divert': 354,\n",
       " 'do': 355,\n",
       " 'doesn': 356,\n",
       " 'doing': 357,\n",
       " 'domestic': 358,\n",
       " 'don': 359,\n",
       " 'done': 360,\n",
       " 'donkey': 361,\n",
       " 'down': 362,\n",
       " 'dozen': 363,\n",
       " 'dragged': 364,\n",
       " 'drawing-room': 365,\n",
       " 'drawing-rooms': 366,\n",
       " 'drawn': 367,\n",
       " 'dress-closets': 368,\n",
       " 'drew': 369,\n",
       " 'dropped': 370,\n",
       " 'each': 371,\n",
       " 'earth': 372,\n",
       " 'ease': 373,\n",
       " 'easel': 374,\n",
       " 'easy': 375,\n",
       " 'echoed': 376,\n",
       " 'economy': 377,\n",
       " 'effect': 378,\n",
       " 'effects': 379,\n",
       " 'efforts': 380,\n",
       " 'egregious': 381,\n",
       " 'eighteenth-century': 382,\n",
       " 'elbow': 383,\n",
       " 'elegant': 384,\n",
       " 'else': 385,\n",
       " 'embarrassed': 386,\n",
       " 'enabled': 387,\n",
       " 'end': 388,\n",
       " 'endless': 389,\n",
       " 'enjoy': 390,\n",
       " 'enlightenment': 391,\n",
       " 'enough': 392,\n",
       " 'ensuing': 393,\n",
       " 'equally': 394,\n",
       " 'equanimity': 395,\n",
       " 'escape': 396,\n",
       " 'established': 397,\n",
       " 'etching': 398,\n",
       " 'even': 399,\n",
       " 'event': 400,\n",
       " 'ever': 401,\n",
       " 'everlasting': 402,\n",
       " 'every': 403,\n",
       " 'exasperated': 404,\n",
       " 'except': 405,\n",
       " 'excuse': 406,\n",
       " 'excusing': 407,\n",
       " 'existed': 408,\n",
       " 'expected': 409,\n",
       " 'exquisite': 410,\n",
       " 'exquisitely': 411,\n",
       " 'extenuation': 412,\n",
       " 'exterminating': 413,\n",
       " 'extracting': 414,\n",
       " 'eye': 415,\n",
       " 'eyebrows': 416,\n",
       " 'eyes': 417,\n",
       " 'face': 418,\n",
       " 'faces': 419,\n",
       " 'fact': 420,\n",
       " 'faded': 421,\n",
       " 'failed': 422,\n",
       " 'failure': 423,\n",
       " 'fair': 424,\n",
       " 'faith': 425,\n",
       " 'false': 426,\n",
       " 'familiar': 427,\n",
       " 'famille-verte': 428,\n",
       " 'fancy': 429,\n",
       " 'fashionable': 430,\n",
       " 'fate': 431,\n",
       " 'feather': 432,\n",
       " 'feet': 433,\n",
       " 'fell': 434,\n",
       " 'fellow': 435,\n",
       " 'felt': 436,\n",
       " 'few': 437,\n",
       " 'fewer': 438,\n",
       " 'finality': 439,\n",
       " 'find': 440,\n",
       " 'fingers': 441,\n",
       " 'first': 442,\n",
       " 'fit': 443,\n",
       " 'fitting': 444,\n",
       " 'five': 445,\n",
       " 'flash': 446,\n",
       " 'flashed': 447,\n",
       " 'florid': 448,\n",
       " 'flowers': 449,\n",
       " 'fluently': 450,\n",
       " 'flung': 451,\n",
       " 'follow': 452,\n",
       " 'followed': 453,\n",
       " 'fond': 454,\n",
       " 'footstep': 455,\n",
       " 'for': 456,\n",
       " 'forced': 457,\n",
       " 'forcing': 458,\n",
       " 'forehead': 459,\n",
       " 'foreign': 460,\n",
       " 'foreseen': 461,\n",
       " 'forgive': 462,\n",
       " 'forgotten': 463,\n",
       " 'form': 464,\n",
       " 'formed': 465,\n",
       " 'forming': 466,\n",
       " 'forward': 467,\n",
       " 'fostered': 468,\n",
       " 'found': 469,\n",
       " 'foundations': 470,\n",
       " 'fragment': 471,\n",
       " 'fragments': 472,\n",
       " 'frame': 473,\n",
       " 'frames': 474,\n",
       " 'frequently': 475,\n",
       " 'friend': 476,\n",
       " 'from': 477,\n",
       " 'full': 478,\n",
       " 'fullest': 479,\n",
       " 'furiously': 480,\n",
       " 'furrowed': 481,\n",
       " 'garlanded': 482,\n",
       " 'garlands': 483,\n",
       " 'gave': 484,\n",
       " 'genial': 485,\n",
       " 'genius': 486,\n",
       " 'gesture': 487,\n",
       " 'get': 488,\n",
       " 'getting': 489,\n",
       " 'give': 490,\n",
       " 'given': 491,\n",
       " 'glad': 492,\n",
       " 'glanced': 493,\n",
       " 'glimpse': 494,\n",
       " 'gloried': 495,\n",
       " 'glory': 496,\n",
       " 'go': 497,\n",
       " 'going': 498,\n",
       " 'gone': 499,\n",
       " 'good': 500,\n",
       " 'good-breeding': 501,\n",
       " 'good-humoured': 502,\n",
       " 'got': 503,\n",
       " 'grace': 504,\n",
       " 'gradually': 505,\n",
       " 'gray': 506,\n",
       " 'grayish': 507,\n",
       " 'great': 508,\n",
       " 'greatest': 509,\n",
       " 'greatness': 510,\n",
       " 'grew': 511,\n",
       " 'groping': 512,\n",
       " 'growing': 513,\n",
       " 'had': 514,\n",
       " 'hadn': 515,\n",
       " 'hair': 516,\n",
       " 'half': 517,\n",
       " 'half-light': 518,\n",
       " 'half-mechanically': 519,\n",
       " 'hall': 520,\n",
       " 'hand': 521,\n",
       " 'hands': 522,\n",
       " 'handsome': 523,\n",
       " 'hanging': 524,\n",
       " 'happen': 525,\n",
       " 'happened': 526,\n",
       " 'hard': 527,\n",
       " 'hardly': 528,\n",
       " 'has': 529,\n",
       " 'have': 530,\n",
       " 'haven': 531,\n",
       " 'having': 532,\n",
       " 'he': 533,\n",
       " 'head': 534,\n",
       " 'hear': 535,\n",
       " 'heard': 536,\n",
       " 'heart': 537,\n",
       " 'height': 538,\n",
       " 'her': 539,\n",
       " 'here': 540,\n",
       " 'hermit': 541,\n",
       " 'herself': 542,\n",
       " 'hesitations': 543,\n",
       " 'hide': 544,\n",
       " 'high': 545,\n",
       " 'him': 546,\n",
       " 'himself': 547,\n",
       " 'hint': 548,\n",
       " 'his': 549,\n",
       " 'history': 550,\n",
       " 'holding': 551,\n",
       " 'home': 552,\n",
       " 'honour': 553,\n",
       " 'hooded': 554,\n",
       " 'hostess': 555,\n",
       " 'hot-house': 556,\n",
       " 'hour': 557,\n",
       " 'hours': 558,\n",
       " 'house': 559,\n",
       " 'how': 560,\n",
       " 'hung': 561,\n",
       " 'husband': 562,\n",
       " 'idea': 563,\n",
       " 'idle': 564,\n",
       " 'idling': 565,\n",
       " 'if': 566,\n",
       " 'immediately': 567,\n",
       " 'in': 568,\n",
       " 'incense': 569,\n",
       " 'indifferent': 570,\n",
       " 'inevitable': 571,\n",
       " 'inevitably': 572,\n",
       " 'inflexible': 573,\n",
       " 'insensible': 574,\n",
       " 'insignificant': 575,\n",
       " 'instinctively': 576,\n",
       " 'instructive': 577,\n",
       " 'interesting': 578,\n",
       " 'into': 579,\n",
       " 'ironic': 580,\n",
       " 'irony': 581,\n",
       " 'irrelevance': 582,\n",
       " 'irrevocable': 583,\n",
       " 'is': 584,\n",
       " 'it': 585,\n",
       " 'its': 586,\n",
       " 'itself': 587,\n",
       " 'jardiniere': 588,\n",
       " 'jealousy': 589,\n",
       " 'just': 590,\n",
       " 'keep': 591,\n",
       " 'kept': 592,\n",
       " 'kind': 593,\n",
       " 'knees': 594,\n",
       " 'knew': 595,\n",
       " 'know': 596,\n",
       " 'known': 597,\n",
       " 'laid': 598,\n",
       " 'lair': 599,\n",
       " 'landing': 600,\n",
       " 'language': 601,\n",
       " 'last': 602,\n",
       " 'late': 603,\n",
       " 'later': 604,\n",
       " 'latter': 605,\n",
       " 'laugh': 606,\n",
       " 'laughed': 607,\n",
       " 'lay': 608,\n",
       " 'leading': 609,\n",
       " 'lean': 610,\n",
       " 'learned': 611,\n",
       " 'least': 612,\n",
       " 'leathery': 613,\n",
       " 'leave': 614,\n",
       " 'led': 615,\n",
       " 'left': 616,\n",
       " 'leisure': 617,\n",
       " 'lends': 618,\n",
       " 'lent': 619,\n",
       " 'let': 620,\n",
       " 'lies': 621,\n",
       " 'life': 622,\n",
       " 'life-likeness': 623,\n",
       " 'lift': 624,\n",
       " 'lifted': 625,\n",
       " 'light': 626,\n",
       " 'lightly': 627,\n",
       " 'like': 628,\n",
       " 'liked': 629,\n",
       " 'line': 630,\n",
       " 'lines': 631,\n",
       " 'lingered': 632,\n",
       " 'lips': 633,\n",
       " 'lit': 634,\n",
       " 'little': 635,\n",
       " 'live': 636,\n",
       " 'll': 637,\n",
       " 'loathing': 638,\n",
       " 'long': 639,\n",
       " 'longed': 640,\n",
       " 'longer': 641,\n",
       " 'look': 642,\n",
       " 'looked': 643,\n",
       " 'looking': 644,\n",
       " 'lose': 645,\n",
       " 'loss': 646,\n",
       " 'lounging': 647,\n",
       " 'lovely': 648,\n",
       " 'lucky': 649,\n",
       " 'lump': 650,\n",
       " 'luncheon-table': 651,\n",
       " 'luxury': 652,\n",
       " 'lying': 653,\n",
       " 'made': 654,\n",
       " 'make': 655,\n",
       " 'man': 656,\n",
       " 'manage': 657,\n",
       " 'managed': 658,\n",
       " 'mantel-piece': 659,\n",
       " 'marble': 660,\n",
       " 'married': 661,\n",
       " 'may': 662,\n",
       " 'me': 663,\n",
       " 'meant': 664,\n",
       " 'mediocrity': 665,\n",
       " 'medium': 666,\n",
       " 'mentioned': 667,\n",
       " 'mere': 668,\n",
       " 'merely': 669,\n",
       " 'met': 670,\n",
       " 'might': 671,\n",
       " 'mighty': 672,\n",
       " 'millionaire': 673,\n",
       " 'mine': 674,\n",
       " 'minute': 675,\n",
       " 'minutes': 676,\n",
       " 'mirrors': 677,\n",
       " 'modest': 678,\n",
       " 'modesty': 679,\n",
       " 'moment': 680,\n",
       " 'money': 681,\n",
       " 'monumental': 682,\n",
       " 'mood': 683,\n",
       " 'morbidly': 684,\n",
       " 'more': 685,\n",
       " 'most': 686,\n",
       " 'mourn': 687,\n",
       " 'mourned': 688,\n",
       " 'moustache': 689,\n",
       " 'moved': 690,\n",
       " 'much': 691,\n",
       " 'muddling': 692,\n",
       " 'multiplied': 693,\n",
       " 'murmur': 694,\n",
       " 'muscles': 695,\n",
       " 'must': 696,\n",
       " 'my': 697,\n",
       " 'myself': 698,\n",
       " 'mysterious': 699,\n",
       " 'naive': 700,\n",
       " 'near': 701,\n",
       " 'nearly': 702,\n",
       " 'negatived': 703,\n",
       " 'nervous': 704,\n",
       " 'nervousness': 705,\n",
       " 'neutral': 706,\n",
       " 'never': 707,\n",
       " 'next': 708,\n",
       " 'no': 709,\n",
       " 'none': 710,\n",
       " 'not': 711,\n",
       " 'note': 712,\n",
       " 'nothing': 713,\n",
       " 'now': 714,\n",
       " 'nymphs': 715,\n",
       " 'oak': 716,\n",
       " 'obituary': 717,\n",
       " 'object': 718,\n",
       " 'objects': 719,\n",
       " 'occurred': 720,\n",
       " 'oddly': 721,\n",
       " 'of': 722,\n",
       " 'off': 723,\n",
       " 'often': 724,\n",
       " 'oh': 725,\n",
       " 'old': 726,\n",
       " 'on': 727,\n",
       " 'once': 728,\n",
       " 'one': 729,\n",
       " 'ones': 730,\n",
       " 'only': 731,\n",
       " 'onto': 732,\n",
       " 'open': 733,\n",
       " 'or': 734,\n",
       " 'other': 735,\n",
       " 'our': 736,\n",
       " 'ourselves': 737,\n",
       " 'out': 738,\n",
       " 'outline': 739,\n",
       " 'oval': 740,\n",
       " 'over': 741,\n",
       " 'own': 742,\n",
       " 'packed': 743,\n",
       " 'paid': 744,\n",
       " 'paint': 745,\n",
       " 'painted': 746,\n",
       " 'painter': 747,\n",
       " 'painting': 748,\n",
       " 'pale': 749,\n",
       " 'paled': 750,\n",
       " 'palm-trees': 751,\n",
       " 'panel': 752,\n",
       " 'panelling': 753,\n",
       " 'pardonable': 754,\n",
       " 'pardoned': 755,\n",
       " 'part': 756,\n",
       " 'passages': 757,\n",
       " 'passing': 758,\n",
       " 'past': 759,\n",
       " 'pastels': 760,\n",
       " 'pathos': 761,\n",
       " 'patient': 762,\n",
       " 'people': 763,\n",
       " 'perceptible': 764,\n",
       " 'perfect': 765,\n",
       " 'persistence': 766,\n",
       " 'persuasively': 767,\n",
       " 'phrase': 768,\n",
       " 'picture': 769,\n",
       " 'pictures': 770,\n",
       " 'pines': 771,\n",
       " 'pink': 772,\n",
       " 'place': 773,\n",
       " 'placed': 774,\n",
       " 'plain': 775,\n",
       " 'platitudes': 776,\n",
       " 'pleased': 777,\n",
       " 'pockets': 778,\n",
       " 'point': 779,\n",
       " 'poised': 780,\n",
       " 'poor': 781,\n",
       " 'portrait': 782,\n",
       " 'posing': 783,\n",
       " 'possessed': 784,\n",
       " 'poverty': 785,\n",
       " 'predicted': 786,\n",
       " 'preliminary': 787,\n",
       " 'presenting': 788,\n",
       " 'prestidigitation': 789,\n",
       " 'pretty': 790,\n",
       " 'previous': 791,\n",
       " 'price': 792,\n",
       " 'pride': 793,\n",
       " 'princely': 794,\n",
       " 'prism': 795,\n",
       " 'problem': 796,\n",
       " 'proclaiming': 797,\n",
       " 'prodigious': 798,\n",
       " 'profusion': 799,\n",
       " 'protest': 800,\n",
       " 'prove': 801,\n",
       " 'public': 802,\n",
       " 'purblind': 803,\n",
       " 'purely': 804,\n",
       " 'pushed': 805,\n",
       " 'put': 806,\n",
       " 'qualities': 807,\n",
       " 'quality': 808,\n",
       " 'queerly': 809,\n",
       " 'question': 810,\n",
       " 'quickly': 811,\n",
       " 'quietly': 812,\n",
       " 'quite': 813,\n",
       " 'quote': 814,\n",
       " 'rain': 815,\n",
       " 'raised': 816,\n",
       " 'random': 817,\n",
       " 'rather': 818,\n",
       " 're': 819,\n",
       " 'real': 820,\n",
       " 'really': 821,\n",
       " 'reared': 822,\n",
       " 'reason': 823,\n",
       " 'reassurance': 824,\n",
       " 'recovering': 825,\n",
       " 'recreated': 826,\n",
       " 'reflected': 827,\n",
       " 'reflection': 828,\n",
       " 'regrets': 829,\n",
       " 'relatively': 830,\n",
       " 'remained': 831,\n",
       " 'remember': 832,\n",
       " 'reminded': 833,\n",
       " 'repeating': 834,\n",
       " 'represented': 835,\n",
       " 'reproduction': 836,\n",
       " 'resented': 837,\n",
       " 'resolve': 838,\n",
       " 'resources': 839,\n",
       " 'rest': 840,\n",
       " 'rich': 841,\n",
       " 'ridiculous': 842,\n",
       " 'robbed': 843,\n",
       " 'romantic': 844,\n",
       " 'room': 845,\n",
       " 'rose': 846,\n",
       " 'rs': 847,\n",
       " 'rule': 848,\n",
       " 'run': 849,\n",
       " 's': 850,\n",
       " 'said': 851,\n",
       " 'same': 852,\n",
       " 'satisfaction': 853,\n",
       " 'savour': 854,\n",
       " 'saw': 855,\n",
       " 'say': 856,\n",
       " 'saying': 857,\n",
       " 'says': 858,\n",
       " 'scorn': 859,\n",
       " 'scornful': 860,\n",
       " 'secret': 861,\n",
       " 'see': 862,\n",
       " 'seemed': 863,\n",
       " 'seen': 864,\n",
       " 'self-confident': 865,\n",
       " 'send': 866,\n",
       " 'sensation': 867,\n",
       " 'sensitive': 868,\n",
       " 'sent': 869,\n",
       " 'serious': 870,\n",
       " 'set': 871,\n",
       " 'sex': 872,\n",
       " 'shade': 873,\n",
       " 'shaking': 874,\n",
       " 'shall': 875,\n",
       " 'she': 876,\n",
       " 'shirked': 877,\n",
       " 'short': 878,\n",
       " 'should': 879,\n",
       " 'shoulder': 880,\n",
       " 'shoulders': 881,\n",
       " 'show': 882,\n",
       " 'showed': 883,\n",
       " 'showy': 884,\n",
       " 'shrug': 885,\n",
       " 'shrugged': 886,\n",
       " 'sight': 887,\n",
       " 'sign': 888,\n",
       " 'silent': 889,\n",
       " 'silver': 890,\n",
       " 'similar': 891,\n",
       " 'simpleton': 892,\n",
       " 'simplifications': 893,\n",
       " 'simply': 894,\n",
       " 'since': 895,\n",
       " 'single': 896,\n",
       " 'sitter': 897,\n",
       " 'sitters': 898,\n",
       " 'sketch': 899,\n",
       " 'skill': 900,\n",
       " 'slight': 901,\n",
       " 'slightly': 902,\n",
       " 'slowly': 903,\n",
       " 'small': 904,\n",
       " 'smile': 905,\n",
       " 'smiling': 906,\n",
       " 'sneer': 907,\n",
       " 'so': 908,\n",
       " 'solace': 909,\n",
       " 'some': 910,\n",
       " 'somebody': 911,\n",
       " 'something': 912,\n",
       " 'spacious': 913,\n",
       " 'spaniel': 914,\n",
       " 'speaking-tubes': 915,\n",
       " 'speculations': 916,\n",
       " 'spite': 917,\n",
       " 'splash': 918,\n",
       " 'square': 919,\n",
       " 'stairs': 920,\n",
       " 'stammer': 921,\n",
       " 'stand': 922,\n",
       " 'standing': 923,\n",
       " 'started': 924,\n",
       " 'stay': 925,\n",
       " 'still': 926,\n",
       " 'stocked': 927,\n",
       " 'stood': 928,\n",
       " 'stopped': 929,\n",
       " 'stopping': 930,\n",
       " 'straddling': 931,\n",
       " 'straight': 932,\n",
       " 'strain': 933,\n",
       " 'straining': 934,\n",
       " 'strange': 935,\n",
       " 'straw': 936,\n",
       " 'stream': 937,\n",
       " 'stroke': 938,\n",
       " 'strokes': 939,\n",
       " 'strolled': 940,\n",
       " 'strongest': 941,\n",
       " 'strongly': 942,\n",
       " 'struck': 943,\n",
       " 'studio': 944,\n",
       " 'stuff': 945,\n",
       " 'subject': 946,\n",
       " 'substantial': 947,\n",
       " 'suburban': 948,\n",
       " 'such': 949,\n",
       " 'suddenly': 950,\n",
       " 'suffered': 951,\n",
       " 'sugar': 952,\n",
       " 'suggested': 953,\n",
       " 'sunburn': 954,\n",
       " 'sunburnt': 955,\n",
       " 'sunlit': 956,\n",
       " 'superb': 957,\n",
       " 'sure': 958,\n",
       " 'surest': 959,\n",
       " 'surface': 960,\n",
       " 'surprise': 961,\n",
       " 'surprised': 962,\n",
       " 'surrounded': 963,\n",
       " 'suspected': 964,\n",
       " 'sweetly': 965,\n",
       " 'sweetness': 966,\n",
       " 'swelling': 967,\n",
       " 'swept': 968,\n",
       " 'swum': 969,\n",
       " 't': 970,\n",
       " 'table': 971,\n",
       " 'take': 972,\n",
       " 'taken': 973,\n",
       " 'talking': 974,\n",
       " 'tea': 975,\n",
       " 'tears': 976,\n",
       " 'technicalities': 977,\n",
       " 'technique': 978,\n",
       " 'tell': 979,\n",
       " 'tells': 980,\n",
       " 'tempting': 981,\n",
       " 'terra-cotta': 982,\n",
       " 'terrace': 983,\n",
       " 'terraces': 984,\n",
       " 'terribly': 985,\n",
       " 'than': 986,\n",
       " 'that': 987,\n",
       " 'the': 988,\n",
       " 'their': 989,\n",
       " 'them': 990,\n",
       " 'then': 991,\n",
       " 'there': 992,\n",
       " 'therefore': 993,\n",
       " 'they': 994,\n",
       " 'thin': 995,\n",
       " 'thing': 996,\n",
       " 'things': 997,\n",
       " 'think': 998,\n",
       " 'this': 999,\n",
       " ...}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4686f356",
   "metadata": {},
   "source": [
    "![](./images/t2id.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c58315fa",
   "metadata": {},
   "source": [
    "- ​现在，我们将所有这些组件整合到一个分词器类（tokenizer class）中"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6b60affc",
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleTokenizerV1:\n",
    "    def __init__(self, vocab):\n",
    "        self.str_to_int = vocab\n",
    "        self.int_to_str = {i: s for s, i in vocab.items()}\n",
    "    \n",
    "    def encode(self, text):\n",
    "        preprocessed = re.split(r'([,.:;?_!\"()\\']|--|\\s)', text)\n",
    "                                \n",
    "        preprocessed = [\n",
    "            item.strip() for item in preprocessed if item.strip()\n",
    "        ]\n",
    "        ids = [self.str_to_int[s] for s in preprocessed]\n",
    "        return ids\n",
    "        \n",
    "    def decode(self, ids):\n",
    "        text = \" \".join([self.int_to_str[i] for i in ids])\n",
    "        text = re.sub(r'\\s+([,.?!\"()\\'])', r'\\1', text)\n",
    "        return text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e875013d",
   "metadata": {},
   "source": [
    "- `encode`函数作用，将文本转换成标识符；\n",
    "- `decode`函数作用，把标识符转换成文本；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d46adc5",
   "metadata": {},
   "source": [
    "- ​我们可以使用分词器（tokenizer）对文本进行编码（即分词与标识化），将其转化为整数序列​​；\n",
    "- ​这些整数（即token ID）后续可通过嵌入层（embedding layers）处理，作为大语言模型（LLM）的输入​；​"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "325601e9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[31, 2, 970, 1126, 401, 311, 1108, 745, 161, 685, 10, 1, 53, 179, 5, 926, 644, 118, 456, 115, 1025, 722, 949, 128, 7]\n"
     ]
    }
   ],
   "source": [
    "tokenizer = SimpleTokenizerV1(vocab)\n",
    "\n",
    "text = \"\"\"Don't you ever dabble with paint any more?\" I asked, still looking about for a trace of such activity.\"\"\"\n",
    "\n",
    "ids = tokenizer.encode(text)\n",
    "\n",
    "print(ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a7d843f1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Don\\' t you ever dabble with paint any more?\" I asked, still looking about for a trace of such activity.'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c146ad1",
   "metadata": {},
   "source": [
    "## 添加特殊的token"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4b6f59a",
   "metadata": {},
   "source": [
    "当我们遇到原始语料库中不存在的单词的时候，我们需要特殊处理。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4be1986b",
   "metadata": {},
   "source": [
    "![](./images/spe-token.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fea6a298",
   "metadata": {},
   "source": [
    "1. ​部分分词器（tokenizer）会使用特殊标记（special tokens）来为大语言模型（LLM）提供额外的上下文信息​​。\n",
    "\n",
    "2. ​常见的特殊标记包括​​：\n",
    "    - ​​[BOS]（序列开始）​​：标记文本的开始。\n",
    "    - [EOS]（序列结束）​​：标记文本的结束。该标记通常用于拼接多个不相关的文本（例如，两篇不同的维基百科文章或两本不同的书籍）。\n",
    "    - ​​[PAD]（填充）​​：当以大于1的批次大小训练LLM时（我们可能会包含多个不同长度的文本），使用填充标记将较短的文本填充至批次中最长的文本长度，以确保所有文本长度一致。\n",
    "    - ​[UNK]（未知）​​：用于表示词汇表（vocabulary）中未包含的词语。\n",
    "\n",
    "3. ​​需要注意的是，GPT-2并不需要上述所有标记，它仅使用一个 <|endoftext|> 标记来降低复杂性​​。\n",
    "    - ​<|endoftext|>​​ 的功能类似于上述的 ​​[EOS]​​ 标记。\n",
    "    - ​​GPT-2 也使用 <|endoftext|> 进行填充​​（由于在批处理输入训练时通常会使用注意力掩码（attention mask），模型本身不会关注被填充的位置，因此这些位置具体是什么标记并不重要）。\n",
    "    - ​GPT-2 不使用 [UNK] 标记来处理未登录词（out-of-vocabulary words）​​，而是采用​​字节对编码（Byte-Pair Encoding, BPE）分词器​​，该分词器会将词汇分解为子词单元（subword units）。我们将在后续章节详细讨论这一点。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81606919",
   "metadata": {},
   "source": [
    "![](./images/eot.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "c9c42190",
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'Hello'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[21], line 3\u001b[0m\n\u001b[1;32m      1\u001b[0m text \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mHello, gencher! Do you like llm?\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m----> 3\u001b[0m \u001b[43mtokenizer\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mencode\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtext\u001b[49m\u001b[43m)\u001b[49m\n",
      "Cell \u001b[0;32mIn[12], line 12\u001b[0m, in \u001b[0;36mSimpleTokenizerV1.encode\u001b[0;34m(self, text)\u001b[0m\n\u001b[1;32m      7\u001b[0m preprocessed \u001b[38;5;241m=\u001b[39m re\u001b[38;5;241m.\u001b[39msplit(\u001b[38;5;124mr\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m([,.:;?_!\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m()\u001b[39m\u001b[38;5;130;01m\\'\u001b[39;00m\u001b[38;5;124m]|--|\u001b[39m\u001b[38;5;124m\\\u001b[39m\u001b[38;5;124ms)\u001b[39m\u001b[38;5;124m'\u001b[39m, text)\n\u001b[1;32m      9\u001b[0m preprocessed \u001b[38;5;241m=\u001b[39m [\n\u001b[1;32m     10\u001b[0m     item\u001b[38;5;241m.\u001b[39mstrip() \u001b[38;5;28;01mfor\u001b[39;00m item \u001b[38;5;129;01min\u001b[39;00m preprocessed \u001b[38;5;28;01mif\u001b[39;00m item\u001b[38;5;241m.\u001b[39mstrip()\n\u001b[1;32m     11\u001b[0m ]\n\u001b[0;32m---> 12\u001b[0m ids \u001b[38;5;241m=\u001b[39m [\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mstr_to_int[s] \u001b[38;5;28;01mfor\u001b[39;00m s \u001b[38;5;129;01min\u001b[39;00m preprocessed]\n\u001b[1;32m     13\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m ids\n",
      "Cell \u001b[0;32mIn[12], line 12\u001b[0m, in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m      7\u001b[0m preprocessed \u001b[38;5;241m=\u001b[39m re\u001b[38;5;241m.\u001b[39msplit(\u001b[38;5;124mr\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m([,.:;?_!\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m()\u001b[39m\u001b[38;5;130;01m\\'\u001b[39;00m\u001b[38;5;124m]|--|\u001b[39m\u001b[38;5;124m\\\u001b[39m\u001b[38;5;124ms)\u001b[39m\u001b[38;5;124m'\u001b[39m, text)\n\u001b[1;32m      9\u001b[0m preprocessed \u001b[38;5;241m=\u001b[39m [\n\u001b[1;32m     10\u001b[0m     item\u001b[38;5;241m.\u001b[39mstrip() \u001b[38;5;28;01mfor\u001b[39;00m item \u001b[38;5;129;01min\u001b[39;00m preprocessed \u001b[38;5;28;01mif\u001b[39;00m item\u001b[38;5;241m.\u001b[39mstrip()\n\u001b[1;32m     11\u001b[0m ]\n\u001b[0;32m---> 12\u001b[0m ids \u001b[38;5;241m=\u001b[39m [\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mstr_to_int\u001b[49m\u001b[43m[\u001b[49m\u001b[43ms\u001b[49m\u001b[43m]\u001b[49m \u001b[38;5;28;01mfor\u001b[39;00m s \u001b[38;5;129;01min\u001b[39;00m preprocessed]\n\u001b[1;32m     13\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m ids\n",
      "\u001b[0;31mKeyError\u001b[0m: 'Hello'"
     ]
    }
   ],
   "source": [
    "text = \"Hello, gencher! Do you like llm?\"\n",
    "\n",
    "tokenizer.encode(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "269e9e1d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1132"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_tokens = sorted(list(set(processed)))\n",
    "\n",
    "all_tokens.extend([\"<|endoftext|>\", \"<|unk|>\"])\n",
    "\n",
    "len(all_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c4ef50a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = {token: integer for integer, token in enumerate(all_tokens)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "71060f0c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('younger', 1127),\n",
       " ('your', 1128),\n",
       " ('yourself', 1129),\n",
       " ('<|endoftext|>', 1130),\n",
       " ('<|unk|>', 1131)]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(vocab.items())[-5:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "49274de3",
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleTokenizerV2:\n",
    "    def __init__(self, vocab):\n",
    "        self.str_to_int = vocab\n",
    "        self.int_to_str = { i:s for s,i in vocab.items()}\n",
    "    \n",
    "    def encode(self, text):\n",
    "        preprocessed = re.split(r'([,.:;?_!\"()\\']|--|\\s)', text)\n",
    "        preprocessed = [item.strip() for item in preprocessed if item.strip()]\n",
    "        preprocessed = [\n",
    "            item if item in self.str_to_int \n",
    "            else \"<|unk|>\" for item in preprocessed\n",
    "        ]\n",
    "\n",
    "        ids = [self.str_to_int[s] for s in preprocessed]\n",
    "        return ids\n",
    "        \n",
    "    def decode(self, ids):\n",
    "        text = \" \".join([self.int_to_str[i] for i in ids])\n",
    "        text = re.sub(r'\\s+([,.:;?!\"()\\'])', r'\\1', text)\n",
    "        return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "fde2ef80",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1131, 5, 1131, 0, 1131, 1126, 628, 1131, 10]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = SimpleTokenizerV2(vocab)\n",
    "\n",
    "tokenizer.encode(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "4e908776",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<|unk|>, <|unk|>! <|unk|> you like <|unk|>?'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(tokenizer.encode(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f026206c",
   "metadata": {},
   "source": [
    "## 字节对编码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3949e92",
   "metadata": {},
   "source": [
    "- GPT-2 采用了​​字节对编码（Byte Pair Encoding, BPE）​​ 作为其分词器（Tokenizer）。\n",
    "- 这种分词方法能够将​​未在预定义词汇表中出现的单词​​分解成更小的​​子词单元​​（subword units），甚至分解到​​字符级别​​，从而使模型具备处理​​未登录词​​（Out-of-Vocabulary, OOV）的能力，https://tiktokenizer.vercel.app/。\n",
    "- 例如，如果 GPT-2 的词表中没有“unfamiliarword”这个词，它可能会根据训练所得的 BPE 合并规则，将其分解为诸如 [\"unfam\", \"iliar\", \"word\"] 或其他子词组合的形式。\n",
    "- OpenAI 开源的 BPE 分词器原始实现可在以下链接找到：https://github.com/openai/gpt-2/blob/master/src/encoder.py。\n",
    "- 而在本章中，我们使用的是 OpenAI 开源库 tiktoken 中的 BPE 分词器，该库使用 ​​Rust​​ 语言实现了 BPE 核心算法，显著提升了​​计算性能​​。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "8bd10974",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.9.0'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tiktoken\n",
    "\n",
    "tiktoken.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "8cefc54e",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = tiktoken.get_encoding(\"gpt2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "69b27d8f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[15496, 11, 2429, 2044, 0, 2141, 345, 588, 32660, 76, 30]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"Hello, gencher! Do you like llm?\"\n",
    "\n",
    "tokenizer.encode(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "3e98bbee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hello, gencher! Do you like llm?'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(tokenizer.encode(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "191aacfc",
   "metadata": {},
   "source": [
    "## ​滑动窗口数据采样"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9a23172",
   "metadata": {},
   "source": [
    "我们以大语言模型逐词生成的方式对其进行训练，因此需要相应地准备训练数据，使序列中的下一个词作为待预测的目标。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b7b3613",
   "metadata": {},
   "source": [
    "![](./images/llm-predict.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "f34ea445",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5145\n"
     ]
    }
   ],
   "source": [
    "with open(\"corpus.txt\", \"r\", encoding=\"utf-8\") as f:\n",
    "    raw_txt = f.read()\n",
    "\n",
    "enc_text = tokenizer.encode(raw_txt)\n",
    "print(len(enc_text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "9aa6afc4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5095"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enc_sample = enc_text[50:]\n",
    "len(enc_sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "720caef1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: [290, 4920, 2241, 287]\n",
      "y: [4920, 2241, 287, 257]\n"
     ]
    }
   ],
   "source": [
    "context_size = 4\n",
    "\n",
    "x = enc_sample[:context_size]\n",
    "y = enc_sample[1:context_size+1]\n",
    "\n",
    "print(f\"x: {x}\")\n",
    "print(f\"y: {y}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a310550e",
   "metadata": {},
   "source": [
    "- 对于每个文本块（text chunk），我们需要明确其​​输入（inputs）​​ 和​​目标（targets）​​。\n",
    "- 由于我们的目标是让模型预测下一个词（next word），因此​​目标（targets）​​ 实际上是​​输入（inputs）​​ 向右移动一个位置（shifted by one position to the right）后的结果。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "692ec12c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " and ----->  established\n",
      " and established ----->  himself\n",
      " and established himself ----->  in\n",
      " and established himself in ----->  a\n"
     ]
    }
   ],
   "source": [
    "for i in range(1, context_size+1):\n",
    "    context = enc_sample[:i]\n",
    "    target = enc_sample[i]\n",
    "\n",
    "    print(tokenizer.decode(context), \"----->\", tokenizer.decode([target]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "d79e9a13",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56b0011c",
   "metadata": {},
   "source": [
    "- 我们将在后续章节涵盖​​注意力机制（Attention Mechanism）​​后，再处理​​下一词预测（next-word prediction）​​的问题。\n",
    "- 目前，我们实现一个​​简单的数据加载器（simple data loader）​​，它能够​​迭代（iterates over）​​输入数据集，并返回输入及与之错位一位的目标数据（即输入序列中每个位置对应的下一词作为目标）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32b95b8d",
   "metadata": {},
   "source": [
    "![](./images/slide-window.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "4740f4b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset, DataLoader\n",
    "\n",
    "\n",
    "class GPTDatasetV1(Dataset):\n",
    "    def __init__(self, txt, tokenizer, max_length, stride):\n",
    "        self.input_ids = []\n",
    "        self.target_ids = []\n",
    "\n",
    "        # Tokenize the entire text\n",
    "        token_ids = tokenizer.encode(txt, allowed_special={\"<|endoftext|>\"})\n",
    "        assert len(token_ids) > max_length, \"Number of tokenized inputs must at least be equal to max_length+1\"\n",
    "\n",
    "        # Use a sliding window to chunk the book into overlapping sequences of max_length\n",
    "        for i in range(0, len(token_ids)-max_length, stride):\n",
    "            input_chunk = token_ids[i: i+max_length]\n",
    "            target_chunk = token_ids[i+1: i+max_length+1]\n",
    "            self.input_ids.append(torch.tensor(input_chunk))\n",
    "            self.target_ids.append(torch.tensor(target_chunk))\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.input_ids)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        return self.input_ids[idx], self.target_ids[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "acd812a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_dataloader_v1(txt, batch_size=4, max_length=256, \n",
    "                         stride=128, shuffle=True, drop_last=True,\n",
    "                         num_workers=0):\n",
    "\n",
    "    # Initialize the tokenizer\n",
    "    tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "\n",
    "    # Create dataset\n",
    "    dataset = GPTDatasetV1(txt, tokenizer, max_length, stride)\n",
    "\n",
    "    # Create dataloader\n",
    "    dataloader = DataLoader(\n",
    "        dataset,\n",
    "        batch_size=batch_size,\n",
    "        shuffle=shuffle,\n",
    "        drop_last=drop_last,\n",
    "        num_workers=num_workers\n",
    "    )\n",
    "\n",
    "    return dataloader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "37848468",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"corpus.txt\", \"r\", encoding=\"utf-8\") as f:\n",
    "    raw_txt = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "c06c0240",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[tensor([[  40,  367, 2885, 1464]]), tensor([[ 367, 2885, 1464, 1807]])]\n"
     ]
    }
   ],
   "source": [
    "dataloader = create_dataloader_v1(raw_txt, batch_size=1, max_length=4, stride=4, shuffle=False)\n",
    "\n",
    "data_iter = iter(dataloader)\n",
    "first_batch = next(data_iter)\n",
    "\n",
    "print(first_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "539c93d2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[tensor([[1807, 3619,  402,  271]]), tensor([[ 3619,   402,   271, 10899]])]\n"
     ]
    }
   ],
   "source": [
    "second_batch = next(data_iter)\n",
    "print(second_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "23ea0cb2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inputs:\n",
      " tensor([[   40,   367,  2885,  1464],\n",
      "        [ 1807,  3619,   402,   271],\n",
      "        [10899,  2138,   257,  7026],\n",
      "        [15632,   438,  2016,   257],\n",
      "        [  922,  5891,  1576,   438],\n",
      "        [  568,   340,   373,   645],\n",
      "        [ 1049,  5975,   284,   502],\n",
      "        [  284,  3285,   326,    11]])\n",
      "\n",
      "Targets:\n",
      " tensor([[  367,  2885,  1464,  1807],\n",
      "        [ 3619,   402,   271, 10899],\n",
      "        [ 2138,   257,  7026, 15632],\n",
      "        [  438,  2016,   257,   922],\n",
      "        [ 5891,  1576,   438,   568],\n",
      "        [  340,   373,   645,  1049],\n",
      "        [ 5975,   284,   502,   284],\n",
      "        [ 3285,   326,    11,   287]])\n"
     ]
    }
   ],
   "source": [
    "dataloader = create_dataloader_v1(raw_txt, batch_size=8, max_length=4, stride=4, shuffle=False)\n",
    "\n",
    "data_iter = iter(dataloader)\n",
    "inputs, targets = next(data_iter)\n",
    "print(\"Inputs:\\n\", inputs)\n",
    "print(\"\\nTargets:\\n\", targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f9eeee6",
   "metadata": {},
   "source": [
    "## ​创建 token 嵌入（Creating token embeddings）​​\n",
    "\n",
    "- 数据已为 LLM 准备就绪；\n",
    "- 最后，我们使用一个​​嵌入层（embedding layer）​​ 将这些 token 嵌入到​​连续向量表示（continuous vector representation）​​ 中；\n",
    "- 通常，这些嵌入层​​本身就是 LLM 的一部分​​，并在模型训练过程中​​被更新（更新权重）；"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cff87afe",
   "metadata": {},
   "source": [
    "![](./images/embedding-framework.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "87415ae6",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_ids = torch.tensor([2, 3, 5, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77217c19",
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size = 6\n",
    "output_dim = 3\n",
    "\n",
    "torch.manual_seed(1)\n",
    "embedding_layer = torch.nn.Embedding(vocab_size, output_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "23f49183",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameter containing:\n",
      "tensor([[-1.5256, -0.7502,  0.6995],\n",
      "        [ 0.1991,  0.8657,  0.2444],\n",
      "        [-0.6629,  0.8073,  0.4391],\n",
      "        [ 1.1712, -2.2456, -1.4465],\n",
      "        [ 0.0612, -0.6177, -0.7981],\n",
      "        [-0.1316, -0.7984,  0.3357]], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "print(embedding_layer.weight)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "9956868f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.6629,  0.8073,  0.4391]], grad_fn=<EmbeddingBackward0>)\n"
     ]
    }
   ],
   "source": [
    "print(embedding_layer(torch.tensor([2])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "643238da",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.6629,  0.8073,  0.4391],\n",
      "        [ 1.1712, -2.2456, -1.4465],\n",
      "        [-0.1316, -0.7984,  0.3357],\n",
      "        [ 0.1991,  0.8657,  0.2444]], grad_fn=<EmbeddingBackward0>)\n"
     ]
    }
   ],
   "source": [
    "print(embedding_layer(input_ids))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab7cac54",
   "metadata": {},
   "source": [
    "## 位置信息编码\n",
    "\n",
    "嵌入层（Embedding layer）会将 ID 转换为相同的向量表示（identical vector representations），而无论这些 ID 在输入序列（input sequence）中处于何种位置​​："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "211ae0cc",
   "metadata": {},
   "source": [
    "![](./images/position.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a40690f",
   "metadata": {},
   "source": [
    "​位置嵌入（Positional embeddings）会与词嵌入向量（token embedding vector）相结合，共同构成大语言模型（LLM）的输入嵌入（input embeddings）："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18f0d641",
   "metadata": {},
   "source": [
    "![](./images/pos-exam.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "53af34e9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50257"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.n_vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "fa41f492",
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size = tokenizer.n_vocab\n",
    "output_dim = 256\n",
    "\n",
    "token_embedding_layer = torch.nn.Embedding(vocab_size, output_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "8eae201e",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataloader = create_dataloader_v1(raw_txt, batch_size=8, max_length=4, stride=4, shuffle=False)\n",
    "\n",
    "data_iter = iter(dataloader)\n",
    "inputs, targets = next(data_iter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "f4e2ab5f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[   40,   367,  2885,  1464],\n",
      "        [ 1807,  3619,   402,   271],\n",
      "        [10899,  2138,   257,  7026],\n",
      "        [15632,   438,  2016,   257],\n",
      "        [  922,  5891,  1576,   438],\n",
      "        [  568,   340,   373,   645],\n",
      "        [ 1049,  5975,   284,   502],\n",
      "        [  284,  3285,   326,    11]])\n",
      "torch.Size([8, 4])\n"
     ]
    }
   ],
   "source": [
    "print(inputs)\n",
    "print(inputs.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "d145ea37",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([8, 4, 256])\n"
     ]
    }
   ],
   "source": [
    "token_embedding = token_embedding_layer(inputs)\n",
    "\n",
    "print(token_embedding.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "22dba3df",
   "metadata": {},
   "outputs": [],
   "source": [
    "context_length = 4\n",
    "\n",
    "pos_embedding_layer = torch.nn.Embedding(context_length, output_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "c6ed02c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_embeddings = pos_embedding_layer(torch.arange(context_length))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "8d9682ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([4, 256])\n"
     ]
    }
   ],
   "source": [
    "print(pos_embeddings.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "00303384",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.2613,  0.3256, -0.4454,  ...,  3.0455, -0.0798,  2.9932],\n",
       "        [-0.6648, -0.2337,  0.8968,  ..., -0.8002,  0.7163,  0.6443],\n",
       "        [-3.6261,  0.5516,  1.2376,  ..., -0.4636,  1.9044, -1.4771],\n",
       "        [-0.5670, -2.0330,  0.6860,  ...,  1.5316,  1.1164, -1.4996]],\n",
       "       grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first = token_embedding[0] + pos_embeddings\n",
    "\n",
    "first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "f7641937",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 256])"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "04b19b9b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[-1.2613,  0.3256, -0.4454,  ...,  3.0455, -0.0798,  2.9932],\n",
       "         [-0.6648, -0.2337,  0.8968,  ..., -0.8002,  0.7163,  0.6443],\n",
       "         [-3.6261,  0.5516,  1.2376,  ..., -0.4636,  1.9044, -1.4771],\n",
       "         [-0.5670, -2.0330,  0.6860,  ...,  1.5316,  1.1164, -1.4996]],\n",
       "\n",
       "        [[-0.0362, -0.5265, -0.8594,  ...,  2.2853,  1.8718,  1.1650],\n",
       "         [-1.1015,  0.2766,  0.5974,  ..., -1.5238,  0.6048,  1.0208],\n",
       "         [-1.3107,  1.5573,  1.7420,  ...,  0.0055,  0.6890, -2.6239],\n",
       "         [ 0.0823, -3.0638, -0.1268,  ..., -0.4628, -0.8048, -0.0914]],\n",
       "\n",
       "        [[-1.0833,  0.3574, -0.7169,  ...,  0.7112,  1.5092,  0.6606],\n",
       "         [-1.5273, -0.7099,  1.2193,  ..., -3.1339, -1.4169,  0.2865],\n",
       "         [-2.0863,  1.0377,  2.2808,  ..., -0.5015,  0.0132, -0.9714],\n",
       "         [ 1.1178, -1.3936, -1.9098,  ..., -2.4585,  0.3691, -0.0640]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[ 0.1007,  2.3258, -0.4064,  ...,  1.0744,  0.4167,  1.0028],\n",
       "         [-1.8863, -0.5999,  0.9602,  ..., -2.9701,  0.9144,  1.0491],\n",
       "         [-1.4857,  1.5364,  0.7422,  ...,  1.1465, -0.9149, -2.1733],\n",
       "         [ 1.0185, -2.9363,  0.2032,  ..., -0.5525,  0.1492, -0.4662]],\n",
       "\n",
       "        [[ 0.5241,  1.0099,  0.8731,  ...,  0.4171,  0.1955,  2.8900],\n",
       "         [-2.1002, -1.8436, -0.0073,  ..., -0.0997, -1.6398,  2.2039],\n",
       "         [-0.5069,  0.8502,  1.1934,  ...,  1.3423,  0.4787, -2.4168],\n",
       "         [ 1.7344, -1.7375, -1.5556,  ..., -0.6855,  2.4026, -1.2368]],\n",
       "\n",
       "        [[ 1.7612,  0.5465, -0.1525,  ...,  2.0832,  1.1157,  0.9650],\n",
       "         [-2.2785,  0.9723,  0.7973,  ..., -3.9412, -1.3274, -1.0440],\n",
       "         [-1.0918, -0.1988,  1.4066,  ..., -1.9927,  0.1266, -2.7237],\n",
       "         [-0.1203, -0.9978, -1.4173,  ..., -1.8602,  1.4910, -1.5055]]],\n",
       "       grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all = token_embedding + pos_embeddings\n",
    "\n",
    "all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5def3068",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5a1b4a82",
   "metadata": {},
   "source": [
    "- 在大语言模型（LLM）输入处理工作流的初始阶段，输入文本首先被分割为独立的词元（tokens）；\n",
    "- 紧接着，这些词元会依据一个预定义的词汇表（vocabulary）被转换为对应的词元ID（token IDs）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b038cff",
   "metadata": {},
   "source": [
    "![](./images/all.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9ff4fa8",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "build-LLM-from-scratch",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
